Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Koos Zevenhoven]

tds333 at gmail.com tds333 at gmail.com writes:

 
 Hi,
 
 still watching progress here. Read all posts and changes.
 
 Everything improved and I know it is a lot of work. Thx for doing this.
 
 But I still think this PEP goes to far.
 
 [...]
 
 We forget to address the major problems here. How can someone in a
 sync script use this async stuff easy. How can async and sync stuff
 cooperate and we don't need to rewrite the world for async stuff.
 How can a normal user access the power of async stuff without rewriting
 all his code. So he can use a simple asyc request library in his code.
 How can a normal user learn and use all this in an easy way.
 
 [...]

Hi Wolfgang,

You may want to see what I just posted on python-ideas. What I wrote about 
is related to several things you mention, and might provide a remedy.

-- Koos



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: Minor API improvements to bytes, bytearray, and memoryview

[Koos Zevenhoven]

On Tue, Jun 7, 2016 at 11:28 PM, Ethan Furman  wrote:
>
> Minor changes: updated version numbers, add punctuation.
>
> The current text seems to take into account Guido's last comments.
>
> Thoughts before asking for acceptance?
>
> PEP: 467
> Title: Minor API improvements for binary sequences
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 2014-03-30
> Python-Version: 3.5
> Post-History: 2014-03-30 2014-08-15 2014-08-16
>
>
> Abstract
> 
>
> During the initial development of the Python 3 language specification, the 
> core ``bytes`` type for arbitrary binary data started as the mutable type 
> that is now referred to as ``bytearray``. Other aspects of operating in the 
> binary domain in Python have also evolved over the course of the Python 3 
> series.
>
> This PEP proposes four small adjustments to the APIs of the ``bytes``, 
> ``bytearray`` and ``memoryview`` types to make it easier to operate entirely 
> in the binary domain:
>
> * Deprecate passing single integer values to ``bytes`` and ``bytearray``
> * Add ``bytes.zeros`` and ``bytearray.zeros`` alternative constructors
> * Add ``bytes.byte`` and ``bytearray.byte`` alternative constructors
> * Add ``bytes.iterbytes``, ``bytearray.iterbytes`` and
>   ``memoryview.iterbytes`` alternative iterators
>

Why not bytes.viewbytes (or whatever name) so that one could also
subscript it? And if it were a property, one could perhaps
conveniently get the n'th byte:

b'abcde'.viewbytes[n]   # compared to b'abcde'[n:n+1]

Also, would it not be more clear to call the int -> bytes method
something like bytes.fromint or bytes.fromord and introduce the same
thing on str? And perhaps allow multiple arguments to create a
str/bytes of length > 1. I guess this may violate TOOWTDI, but anyway,
just a thought.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: Minor API improvements to bytes, bytearray, and memoryview

[Koos Zevenhoven]

On Wed, Jun 8, 2016 at 12:57 AM, Barry Warsaw  wrote:
> On Jun 07, 2016, at 09:40 PM, Brett Cannon wrote:
>
>>On Tue, 7 Jun 2016 at 14:38 Paul Sokolovsky  wrote:
>>> What's wrong with b[i:i+1] ?
>>It always succeeds while indexing can trigger an IndexError.
>
> Right.  You want a method with the semantics of __getitem__() but that returns
> the desired type.
>

And if this is called __getitem__ (with slices delegated to
bytes.__getitem__) and implemented in a class, one has a view. Maybe
I'm missing something, but I fail to understand what makes this
significantly more problematic than an iterator. Ok, I guess we might
also need __len__.

-- Koos

> -Barry
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] proposed os.fspath() change

[Koos Zevenhoven]

On Wed, Jun 15, 2016 at 9:29 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 15 June 2016 at 10:59, Brett Cannon <br...@python.org> wrote:
>>
>>
>> On Wed, 15 Jun 2016 at 09:48 Guido van Rossum <gu...@python.org> wrote:
>>>
>>> These are really two separate proposals.
>>>
>>> I'm okay with checking the return value of calling obj.__fspath__; that's
>>> an error in the object anyways, and it doesn't matter much whether we do
>>> this or not (though when approving the PEP I considered this and decided not
>>> to insert a check for this). But it doesn't affect your example, does it? I
>>> guess it's easier to raise now and change the API in the future to avoid
>>> raising in this case (if we find that raising is undesirable) than the other
>>> way around, so I'm +0 on this.
>>
>> +0 from me as well. I know in some code in the stdlib that has been ported
>> which prior to adding support was explicitly checking for str/bytes this
>> will eliminate its own checking (obviously not a motivating factor as it's
>> pretty minor).
>
> I'd like a strong assertion that the return value of os.fspath() is a
> plausible filesystem path representation (so either bytes or str), and
> *not* some other kind of object that can also be used for accessing
> the filesystem (like a file descriptor or an IO stream)

I agree, so I'm -0.5 on passing through any object (at least by default).

>>> The other proposal (passing anything that's not understood right through)
>>> is more interesting and your use case is somewhat compelling. Catching the
>>> exception coming out of os.fspath() would certainly be much messier. The
>>> question remaining is whether, when this behavior is not desired (e.g. when
>>> the caller of os.fspath() just wants a string that it can pass to open()),
>>> the condition of passing that's neither a string not supports __fspath__
>>> still produces an understandable error. I'm not sure that that's the case.
>>> E.g. open() accepts file descriptors in addition to paths, but I'm not sure
>>> that accepting an integer is a good idea in most cases -- it either gives a
>>> mystery "Bad file descriptor" error or starts reading/writing some random
>>> system file, which it then closes once the stream is closed.
>>
>> The FD issue of magically passing through an int was also a concern when
>> Ethan brought this up in an issue on the tracker. My argument is that FDs
>> are not file paths and so shouldn't magically pass through if we're going to
>> type-check anything or claim os.fspath() only works with paths (FDs are
>> already open file objects). So in my view  either we go ahead and type-check
>> the return value of __fspath__() and thus restrict everything coming out of
>> os.fspath() to Union[str, bytes] or we don't type check anything and be
>> consistent that os.fspath() simply does is call __fspath__() if present.
>>
>> And just  because I'm thinking about it, I would special-case the FDs, not
>> os.PathLike (clearer why you care and faster as it skips the override of
>> __subclasshook__):
>>
>> # Can be a single-line ternary operator if preferred.
>> if not isinstance(filename, int):
>> filename = os.fspath(filename)
>
> Note that the LZMA case Ethan cites is one where the code accepts
> either an already opened file-like object *or* a path-like object, and
> does different things based on which it receives.
>
> In that scenario, rather than introducing an unconditional "filename =
> os.fspath(filename)" before the current logic, it makes more sense to
> me to change the current logic to use the new protocol check rather
> than a strict typecheck on str/bytes:
>
> if isinstance(filename, os.PathLike): # Changed line
> filename = os.fspath(filename)# New line

You are making one of my earlier points here, thanks ;). The point is
that the name PathLike sounds like it would mean anything path-like,
except that os.PathLike does not include str and bytes. And I still
think the naming should be a little different.

So that would be (os.Pathlike, str, bytes) instead of just os.PathLike.

> if "b" not in mode:
> mode += "b"
> self._fp = builtins.open(filename, mode)
> self._closefp = True
> self._mode = mode_code
> elif hasattr(filename, "read") or hasattr(filename, "write"):
> self._fp = filename
> self._mode = mode_code
> else:
> raise TypeError(
>  "filename must be a path-like or file-like object"

Re: [Python-Dev] proposed os.fspath() change

[Koos Zevenhoven]

t;> ...   def __int__(self):
>> ... return 'string'
>> ...   def __index__(self):
>> ... return b'bytestring'
>> ...   def __bool__(self):
>> ... return 'true-ish'
>> ...
>> --> h = Huh()
>>
>> --> int(h)
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> TypeError: __int__ returned non-int (type str)
>>
>> --> ''[h]
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> TypeError: __index__ returned non-int (type bytes)
>>
>> --> bool(h)
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> TypeError: __bool__ should return bool, returned str
>>
>> Arguments in favor or against?
>>
>> --
>> ~Ethan~
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>



-- 
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] proposed os.fspath() change

[Koos Zevenhoven]

On Wed, Jun 15, 2016 at 10:15 PM, Brett Cannon <br...@python.org> wrote:
>
>
> On Wed, 15 Jun 2016 at 12:12 Koos Zevenhoven <k7ho...@gmail.com> wrote:
>>
>> >> if isinstance(filename, os.PathLike):
>>
>> By the way, regarding the line of code above, is there a convention
>> regarding whether implementing some protocol/interface requires
>> registering with (or inheriting from) the appropriate ABC for it to
>> work in all situations. IOW, in this case, is it sufficient to
>> implement __fspath__ to make your type pathlike? Is there a conscious
>> trend towards requiring the ABC?
>
>
> ABCs like os.PathLike can override __subclasshook__ so that registration
> isn't required (see
> https://hg.python.org/cpython/file/default/Lib/os.py#l1136). So registration
> is definitely good to do to be explicit that you're trying to meet an ABC,
> but it isn't strictly required.

Ok I suppose that's fine, so I propose we update the ABC part in the
PEP with __subclasshook__.

And the other question could be turned into whether to make str and
bytes also PathLike in __subclasshook__.

-- Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] proposed os.fspath() change

[Koos Zevenhoven]

>> if isinstance(filename, os.PathLike):

By the way, regarding the line of code above, is there a convention
regarding whether implementing some protocol/interface requires
registering with (or inheriting from) the appropriate ABC for it to
work in all situations. IOW, in this case, is it sufficient to
implement __fspath__ to make your type pathlike? Is there a conscious
trend towards requiring the ABC?

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] proposed os.fspath() change

[Koos Zevenhoven]

On Wed, Jun 15, 2016 at 11:00 PM, Ethan Furman <et...@stoneleaf.us> wrote:
> On 06/15/2016 12:24 PM, Koos Zevenhoven wrote:
>>
>> And the other question could be turned into whether to make str and
>> bytes also PathLike in __subclasshook__.
>
> No, for two reasons.
>
> - most str's and bytes' are not paths;

True. Well, at least most str and bytes objects are not *meant* to be
used as paths, even if they could be.

> - PathLike indicates a rich-path object, which str's and bytes' are not.

This does not count as a reason.

If this were called pathlib.PathABC, I would definitely agree [1]. But
since this is called os.PathLike, I'm not quite as sure. Anyway,
including str and bytes is more of a type hinting issue. And since
type hints will in also act as documentation, the naming of types is
becoming more important.

-- Koos

[1] No, I'm not proposing moving this to pathlib

> --
> ~Ethan~
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Koos Zevenhoven]

On Tue, Apr 12, 2016 at 7:19 PM, Chris Barker  wrote:
>
> One more though came up just now: there are different level sof abstractions
> and representations for paths. We don't want to make Path a subclass of
> string, because Path is supposed to be a higher level abstraction -- good.
>
> then at the bottom of the stack, we NEED the bytes level path, because that
> what ultimately gets passed to the OS.
>
> THe legacy from the single-byte encoding days is that bytes and strings were
> the same, so we could let people work with nice human readable strings,
> while also working with byte paths in the same way -- but those days are
> gone -- py3 make s clear (and important) distiction between nice human
> readable strings  and the bytes that represent them.
>
> So: why use strings as the lingua franca of paths? i.e. the basis of the
> path protocol. maybe we should support only two path representations:
>
> 1) A "proper" path object -- i.e. pathlib.Path or anything else that
> supports the path protocol.
>
> 2) the bytes that the OS actually needs.
>

You do have a point there. But since bytes pathnames are deprecated on
windows, this seems to lead to supporting both str and bytes in the
protocol, or having two protocols __fspathbytes__ and __fspathstr__
(and one being preferred over the other, potentially even depending on
the platform).,

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Koos Zevenhoven]

On Tue, Apr 12, 2016 at 11:56 AM, Nick Coghlan  wrote:
> One possible way to address this concern would be to have the
> underlying protocol be bytes/str (since boundary code frequently needs
> to handle the paths-are-bytes assumption in POSIX), but offer an
> "os.fspathname" API that rejected bytes output from os.fspath. That
> is, it would be equivalent to:
>
> def fspathname(path):
> name = os.fspath(path)
> if not isinstance(name, str):
> raise TypeError("Expected str for pathname, not
> {}".format(type(name)))
> return name
>
> That way folks that wanted the clean "must be str" signature could use
> os.fspathname, while those that wanted to accept either could use the
> lower level os.fspath.

I'm not necessarily opposed to this. I kept bringing up bytes in the
discussion because os.path.* etc. and DirEntry support bytes and will
need to keep doing so for backwards compatibility.  I have no
intention to use bytes pathnames myself. But it may break existing
code if functions, for instance, began to decode bytes paths to str if
they did not previously do so (or to reject them). It is indeed a lot
safer to make new code not support bytes paths than to change the
behavior of old code.

But then again, do we really recommend new code to use os.fspath (or
os.fspathname)? Should they not be using either pathlib or os.path.*
etc. so they don't have to care? I'm sure Ethan and his library (or
some other path library) will manage without the function in the
stdlib, as long as the dunder attribute is there.

So I'm, once again, posing this question (that I don't think got any
reactions previously): Is there a significant audience for this new
function, or is it enough to keep it a private function for the stdlib
to use? That handful of third-party path libraries can decide for
themselves if they want to (a) reject bytes or (b) implicitly fsdecode
them or (c) pass them through just like str, depending on whatever
their case requires in terms of backwards compatiblity or other goals.

If we forget about the os.fswhatever function, we only have to decide
whether the magic dunder attribute can be str or bytes or just str.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Koos Zevenhoven]

On Tue, Apr 12, 2016 at 6:52 PM, Stephen J. Turnbull  wrote:
>
> (A) Why does anybody need bytes out of a pathlib.Path (or other
> __fspath__-toting, higher-level API) *inside* the boundary?  Note
> that the APIs in os (etc) *don't need* bytes because they are
> already polymorphic.
>

Indeed not from pathlib.*Path , but from DirEntry, which may have a
path as bytes. So the options for DirEntry (or things like Ethan's
'antipathy') are:

(1) Provide bytes or str via the protocol, depending on which type
this DirEntry has

Downside: The protocol needs to support str and bytes.

(2) Decode bytes using os.fsdecode and provide a str via the protocol

Downside: The user passed in bytes and maybe had a reason to do so.
This might lead to a weird mixture of str and bytes in the same code.

(3) Do not implement the protocol when dealing with bytes

Downside: If a function calling os.scandir accepts both bytes and str
in a duck-typing fashion, then, if this adopted something that uses
the new protocol, it will lose its bytes compatiblity. This risk might
not be huge, so perhaps (3) is an option?


> (B) If they do, why can't they just apply bytes() to the object?  I
> understand that that would offend Ethan's aesthetic sense, so it's
> worth looking for a nice way around it.  But allowing __fspath__
> to return bytes or str is hideous, because Paths are clearly on
> the application side of the boundary.
>
> Note that bytes() may not have the serious problem that str() does of
> being too catholic about its argument: nothing in __builtins__ has a
> __bytes__!  Of course there are a few things that do work: ints, and
> sequences of ints.

Good point. But this only applies to when the user _explicitly_ deals
with bytes. But when the user just deals with the type (str or bytes)
that is passed in, as os.path.* as well as DirEntry now do, this does
not work.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Koos Zevenhoven]

On Mon, Apr 11, 2016 at 9:27 AM, Nick Coghlan  wrote:
> On 11 April 2016 at 02:16, Ethan Furman  wrote:
>>
>> I guess I don't see the point of this.  Either DirEntry's [1] only get
>> partial support (which is only marginally better than the no support pathlib
>> currently has), or stdlib code will need to catch those errors and then do
>> an isinstance check to see if knows what the type is and how to deal with it
>> [1].
>
> What's wrong with only gaining partial support? Standard library code
> that doesn't currently support DirEntry at all will gain the ability
> to support str-based DirEntry objects, while bytes-based DirEntry
> objects will continue to be a low level object that isn't
> interoperable with most other APIs (which is fine - anyone writing low
> level POSIX-specific code can deal with unpacking the values
> explicitly, it just won't happen implicitly anywhere).
>

While I'm also tempted to lean towards 'marginalizing bytes support',
it seems a little bit dangerous to me. Currently, os.path is heavily
based on duck typing of str and bytes, so there may be code out there
that does all kinds of things with paths without knowing whether it
deals with bytes or str objects. If such code gets in contact with
this pathname protocol, it will raise an exception whenever it happens
to be fed a bytes path. That is, if the approach of 'partial support'
is taken.

And still there is the question I just posted in another branch of
this mess: Who should use os.fspathname(...)? If it's os.path.* and
other traditional (low-level?) functions that deal with paths, then
fspathname should, in the name of backwards compatiblity, be able to
deal with bytes and return bytes in those cases.  Otherwise fspathname
would do nothing for you, and all the work of
isinstance/hasattr/whatever would be left to the caller of
os.fspathname (or maybe this is what you want?).

So a somewhat useful fspathname might indeed look something like this:

 def fspathname(pathlike) -> Union[str, bytes]:
 pathname = getattr(pathlike, '__fspathname__', pathlike)
 if not isinstance(pathname, (str, bytes)):
 raise TypeError("your thing is not pathlike")
 return pathname

But maybe it is enough to have the __fspathname__ attribute, and make
fspathname() some internal implementation detail of os.path.* and the
like.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Koos Zevenhoven]

On Sat, Apr 9, 2016 at 12:53 AM, Brett Cannon <br...@python.org> wrote:
> On Fri, 8 Apr 2016 at 14:23 Koos Zevenhoven <k7ho...@gmail.com> wrote:
>
> At this point no one wants to touch bytes paths. If you need that level of
> control because of multiple encodings within a single file system then you
> will probably have to stick with managing bytes paths on your own to get the
> encoding right.

What does this mean? I assume you don't mean os.path.* would stop
dealing with bytes? And if not, then you seem to mean that os.fspath
would do nothing except call .__fspath__(). In that case, I think we
should go back to it being an attribute (or property) and a variation
of the now very famous idiom getattr(path, '__fspath__', path) and
perhaps have os.fspath do exactly that.

> And just because DirEntry supports bytes doesn't mean that any magic method
> it gains has to carry that forward (it can always raise a TypeError if
> necessary).

No, but what if some code gets pathnames from whatever other places
and passes them on to os.scandir. Whenever it happens to get a bytes
path, a TypeError gets raised, but only when it picks one of the
DirEntry objects and for instance tries to open(...) it. Of course,
I'm not sure how common this is.

> It really depends on how we choose to structure the
> function in terms of just doing the right thing for objects that follow the
> protocol or if we want to introduce some required structure for the
> resulting path and implement some type guarantees so you have a better idea
> of what you will be working with after calling the function.

Do you have an example of potential 'required structure'?

>> Then the options are either to return Union[str, bytes] or to
>> always return str. And if the latter does not cause any problems, I
>> like it way better, and it seems others would do too.
>
> You don't have to convert byte paths to str, you can simply raise an
> exception in the face of them.
>

I thought the point was for existing APIs to start supporting path
objects, wouldn't raising an exception break the API?

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)

[Koos Zevenhoven]

Nick Coghlan wrote:
> On 7 April 2016 at 03:26, Brett Cannon  wrote:
>>
>> Name: __path__, __fspath__, or something else?
>
> __fspath__
>

I think I might like this dunder name because it does not clutter the
list of regular methods and attributes, and is perhaps more pythonic.

>> Method or attribute? (changes what kind of one-liner you might use in
>> libraries, but I think historically all protocols have been methods and the
>> serialized string representation might be costly to build)
>
> Method, as long as there's a helper function somewhere

As a further minor benefit of it being a method, it may be easier to
distinguish it from from `__path__`, which is an iterable attribute.

>> Built-in? (name is dependent on #1 if we add one)
>
> os.fspath (alongside os.fsencode and os.fsdecode)
>
> (Putting this in a module low in the dependency stack makes it easy
> for other modules to access without pulling in all of pathlib's
> dependencies)

Strong +1 on putting it in os. This should also be implemented in
DirEntry, instances of which are "yielded" by os.scandir.

Also, you have a strong case regarding naming with the 'fs' prefix. It
is also easier to read fspath as f-s-path than it is to read ospath as
o-s-path, because ospath could also be pronounced as a single
(meaningless?) word.

I'm still thinking a little bit about 'pathname', which to me sounds
more like a string than fspath does [1]. It would be nice to have the
string/path distinction especially when pathlib adoption grows larger.
But who knows, maybe somewhere in the far future, no-one will care
much about fspath, fsencode, fsdecode or os.path.

>> Add the method/attribute to str? (I assume so, much like __index__() is on
>> int, but I have not seen it explicitly stated so I would rather clarify it)
>
> Makes sense

If added to str, it should also be added to bytes. But will that then
return str or bytes? See also the next point.

> Expand the C API to have something like PyObject_Path()?
>
> PyUnicode_FromFSPath, perhaps? The return type is well-defined here,
> so it can be done as an alternate constructor, and the C API
> counterparts of os.fsdecode and os.fsencode are PyUnicode functions
> (specifically PyUnicode_DecodeFSDefault and PyUnicode_EncodeFSDefault)

What about DirEntry, which may have a bytes representation? I would
expect the function return type of os.fspath to be Union[str, bytes],
unless bytes pathnames are decoded with surrogate escapes.

[1] https://mail.python.org/pipermail/python-ideas/2016-April/039595.html


PS. I have been reading this list occasionally on the google groups
mirror, and I now subscribed to it just to send this. (BTW, I probably
broke the thread, as I did not have Nick's email in my inbox to reply
to. Sorry about that.) I'll have to mention that I was surprised, to
say the least, to find that the pathlib discussion had moved here from
python-ideas, where I had mentioned I was working on a proposal. Then,
I also found that the solution discussed here was seemingly an
improved version of what I had proposed on python-ideas somewhat
earlier [1], but did not get any reactions to. While I can only make
guesses about what happened, these kinds of things easily make you go
from "Hey, maybe I'll be able to do something to improve Python!" to
"These people don't seem to want me here or appreciate my efforts.".
Not to accuse anyone in particular; just to let people know. Anyway, I
somehow got sucked into thinking deeply about pathlib etc. (which I do
use). Not that I really have much at stake here, except spending
ridiculous amounts of time thinking about paths, mainly during my
Easter holidays and after that. I really had a hard time explaining to
friends and family what the heck I was doing ;).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Koos Zevenhoven]

On Fri, Apr 8, 2016 at 8:34 PM, Brett Cannon  wrote:
> On Fri, 8 Apr 2016 at 09:39 Ethan Furman  wrote:
>> > I thought the whole point off all this is that not any old string can be
>> > a path! (whereas any int can be an index). Unless we go with Chris A's
>> > suggestion that this be a more generic lossless string protocol, rather
>> > than just for paths.
>>
>> That does seem to be a valid point against str.__fspath__.
>
> Yep, and I'm expecting we won't want that at this point. The fact that paths
> need strings for low-level OS stuff is a historical and technical detail, so
> no need to drag the entire str type into it if we can provide a reasonable
> helper function (for either the ABC or magic method solution).

I'm not sure I understand what these points are about. Anyway,
disallowing str or bytes as pathnames will break backwards
compatibility if done at some point in the future. There's no way
around that.

But regarding all this talk of mine about bytes is because it has not
been completely clear to me if something can break when converting a
bytes path to str. I did originally propose guaranteeing a str, but I
am so far only 85% convinced that that does not cause any problems. I
understand that fsencode(fsdecode(bytes_path)) should always be equal
to bytes_path. But can some other path operations fail when there are
surrogates in the strings? And again, not to forget DirEntry, which
may have a byte string path.

Either way, I suppose os.fspath should accept anything that has
__fspath__ or is a str or bytes (whether these have the dunder method
or not). Then the options are either to return Union[str, bytes] or to
always return str. And if the latter does not cause any problems, I
like it way better, and it seems others would do too. And in that case
it would probably be time to deprecate bytes paths on posix too (on
Windows, this is already the case).

But do we know that converting all paths to str does not cause any problems?

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)

[Koos Zevenhoven]

On Sat, Apr 9, 2016 at 10:16 AM, Ethan Furman  wrote:
> On 04/09/2016 12:07 AM, Victor Stinner wrote:
>>
>> os.DirEntry doesn't support bytes: os.scandir() only accept str. It's a
>> deliberate choice.
>
>
> 3.5.0 scandir supports bytes:
>
> --> huh = list(scandir(b'.'))
> --> huh
> [, ,  b'__MACOSX'>, , ,  b'index.html'>]
>
> --> huh[0].path
> b'./minicourse-ajax-project'
>
>

Maybe it's the bytes support in scandir that should be deprecated?
(And not bytes support in general, which cannot be done on posix, as I
hear Stephen T. will tell me).

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancments - method name only

[Koos Zevenhoven]

On Sat, Apr 9, 2016 at 10:48 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 9 April 2016 at 04:25, Brett Cannon <br...@python.org> wrote:
>> On Fri, 8 Apr 2016 at 11:13 Ethan Furman <et...@stoneleaf.us> wrote:
>>> On 04/08/2016 10:46 AM, Koos Zevenhoven wrote:
>>>  > On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker  wrote:
>>>  >> On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote:
>>>  >>>
>>>  >>> I'm still thinking a little bit about 'pathname', which to me sounds
>>>  >>> more like a string than fspath does.
>>>  >>
>>>  >>
>>>  >> I like that a lot - or even "__pathstr__" or "__pathstring__"
>>>  >> after all, we're making a big deal out of the fact that a path is
>>>  >> *not a string*, but rather a string is a *representation* (or
>>>  >> serialization) of a path.
>>>
>>> That's a decent point.
>>>
>>> So the plausible choices are, I think:
>>>
>>> - __fspath__  # File System Path -- possible confusion with Path
>>
>> +1
>
> I like __fspath__, but I'm also sympathetic to Koos' point that we're
> really dealing with path *names* being produced via this protocol,
> rather than the paths themselves.
>
> That would bring the completely explicit "__fspathname__" into the
> mix, which would be comparable in length to "__getattribute__" as a
> magic method name (both in terms of number of syllable and number of
> characters).
>
> Considering the helper function usage, here's some examples in
> combination with os.fsencode and os.fsdecode:
>
> # Status quo for binary/text path conversions
> text_path = os.fsdecode(bytes_path)
> bytes_path = os.fsencode(text_path)
>
> # Getting a text path from an arbitrary object
> text_path = os.fspath(obj) # This doesn't scream "returns text!" to me
> text_path = os.fspathname(obj) # This does
>
> # Getting a binary path from an arbitrary object
> bytes_path = os.fsencode(os.fspath(obj))
> bytes_path = os.fsencode(os.fspathname(obj))
>
> I'm starting to think the semantic nudge from the "name" suffix when
> reading the code is worth the extra four characters when writing it
> (keeping in mind that the whole point of this exercise is that most
> folks *won't* be writing explicit conversions - the stdlib will handle
> it on their behalf).
>

Regarding the name, I completely agree with Nick's reasoning (above).
I'm not sure it's a high priority to make dunder-method names short.
They are not typed very often, and when the number of these
"protocols" increases, you face potentially ambiguous names more and
more often (there already is a '__path__' and a '__file__' etc., as
has been brought up earlier in these threads.). In other words, it's a
good idea to have some information in the name.

> I also think the more explicit name helps answer some of the type
> signature questions that have arisen:
>
> 1. Does os.fspathname return rich Path objects? No, it returns names
> as str objects

Or byte strings, it seems, unfortunately.

> 2. Will file descriptors pass through os.fspathname? No, as they're
> not names, they're numeric descriptors.
> 3. Will bytes-like objects pass through os.fspathname? No, as they're
> not names, they're encodings of names
>

If fspathname(...) is to be used in os.path.*, it will break things if
it starts to turn encoded bytes pathnames into str pathnames, which it
did not previously do.

And if fspathname is not to be used in os.path.*, who would be our
intended user of fspathname? I assume we we don't want to encourage
typical 'users' to manipulate pathnames by hand.

>> I personally still like __ospath__ as well.
>
> That one fails the "Is it ambiguous when spoken aloud?" test for me:
> if someone mentions "oh-ess-path", are they talking about os.path or
> __ospath__? With "eff-ess-path" or "eff-ess-path-name", that problem
> doesn't arise.
>

+1 to this too.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)

[Koos Zevenhoven]

On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker <chris.bar...@noaa.gov> wrote:
> On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>>
>> I'm still thinking a little bit about 'pathname', which to me sounds
>> more like a string than fspath does [1].
>
>
> I like that a lot - or even "__pathstr__" or "__pathstring__"
>
> after all, we're making a big deal out of the fact that a path is *not a
> string*, but rather a string is a *representation* (or serialization) of a
> path.

For me, the point here is the reverse: that any str is not a path, and
that it is misleading to call it *path* when whole point is to make it
*not* a specialized path object but a plain string. I think it's ok to
think of a path as special kind of string. For instance, an URI is
explicitly defined as a *sequence of characters*, and URIs can be
thought of as a more recent, improved and broadened concept than
paths. This is the point of view I took in my recent proposal, but I
don't think it's the only valid way to think about paths "in theory".
I like the "serialization" interpretation as well, but i tend to think
that that string serialization is what is called a path.

Anyway, I don't think these philosophical considerations should
dictate how Python is implemented. But it is always good to also have
a valid theoretical point of view to back up a design decision.

> For the record, this is pretty rare -- and it was announced on -ideas that
> the discussion had started up here -- maybe you missed that post?

If you mean in Ethan's response to my proposal, I noticed that, but
the discussions here had already gone quite far by that time. Even
more so by the time I had time to see what was going on.

I do have to say this is not the first time I felt there was some sort
of hostility towards newcomers on python-ideas. Sure, it might be
partly because those people don't know the culture on the list, but
I'm not sure if that should be used as an excuse.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancments - method name only

[Koos Zevenhoven]

On Fri, Apr 8, 2016 at 9:20 PM, Chris Barker  wrote:
>
> we rejected plain old __path__ because this is already ued in another
> context, but if we add "str" on the end, that's not longer an issue, so do
> we need the "fs"?
>
> __pathstr__ # pathstring
>

Or perhaps __pathstring__ in case it may be or return byte strings.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancments - method name only

[Koos Zevenhoven]

On Fri, Apr 8, 2016 at 11:39 PM, R. David Murray <rdmur...@bitdance.com> wrote:
> On Fri, 08 Apr 2016 19:24:44 -, Brett Cannon <br...@python.org> wrote:
>> On Fri, 8 Apr 2016 at 12:10 Chris Angelico <ros...@gmail.com> wrote:
>>
>> > On Sat, Apr 9, 2016 at 5:03 AM, Chris Barker <chris.bar...@noaa.gov>
>> > wrote:
>> > > On Fri, Apr 8, 2016 at 11:34 AM, Koos Zevenhoven <k7ho...@gmail.com>
>> > wrote:
>> > >>
>> > >> >
>> > >> > __pathstr__ # pathstring
>> > >> >
>> > >>
>> > >> Or perhaps __pathstring__ in case it may be or return byte strings.
>
> But there are other paths than OS file system paths.  I prefer
> __fspath__ or __os_path__ myself.  I think the fact that it is a string
> is implied by the fact that it is getting us the thing we can pass
> to the os (since Python3 deals with os paths as strings unless you
> specify otherwise, only converting them back to bytes, on unix, at the last
> moment).
>
> Heh, although I suppose one could make the argument that it should
> return whatever the native OS wants, and save the low level code
> from having to do that?  Pass the path object all the way down
> to that "final step" in the C layer?  (Just ignore me, I'm sure
> I'm only making trouble :)

My favorites are fspath and pathname, and since this is a dunder
methdod, it is not as crucial what it is called. I have the feeling
the consensus is converging towards fspath?

I'll comment on the bytes issue in the other thread. Boy these threads
are all over the place!

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing the provisional label from pathlib

[Koos Zevenhoven]

On Mon, May 23, 2016 at 10:38 PM, Chris Barker  wrote:
> On Mon, May 23, 2016 at 10:13 AM, Brett Cannon  wrote:
>>
>> 3.5 is still getting bugfixes:
>> https://docs.python.org/devguide/#status-of-python-branches
>>
>> As for backporting __fspath__() for pathlib, you can easily write your own
>> subclass that adds it. And since the stdlib won't be updated in 3.5 for
>> consumption of fspath
>
>
> OK -- when I said "it", I meant the whole shebang -- i.e. the stdlib stuff
> too. But fair enough, we can't be backporting everything, and I that would
> be touching a lot of the lib.
>
> -CHB

I guess we might consider adding __fspath__ in maintenance releases,
and make open() support it? That would cover a significant share of
use cases, although it might be weird if code written for 3.5.2
doesn't run on 3.5.1... So maybe just 3.6-> ?

I'm not quite as busy as last week, so I might consider working on the
stdlib changes if I find the time. Or is someone already working on
this?

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Removing the provisional label from pathlib

[Koos Zevenhoven]

On Tue, May 24, 2016 at 4:56 PM, Barry Warsaw <ba...@python.org> wrote:
> On May 24, 2016, at 02:03 PM, Koos Zevenhoven wrote:
>
>>I guess we might consider adding __fspath__ in maintenance releases,
>>and make open() support it? That would cover a significant share of
>>use cases, although it might be weird if code written for 3.5.2
>>doesn't run on 3.5.1...
>
> Please, no.  We learned that lesson in Python 2.2.1 with True/False.

What happened? True was included in 2.2.1 but not False?-). Anyway, I
guess you are probably right, and "3.6->" is the way to go. Besides,
Guido already wrote that in the first response.

-- Koos

> Cheers,
> -Barry
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

**another deep, calming breath**

On Wed, May 11, 2016 at 7:43 PM, Brett Cannon  wrote:
> Open Issues
> ===
>
> Should os.fspath() return bytes?
> 
>

In most cases, it of course should not. The section (or the title) do
not represent my view on the topic, but bytes paths are not going away
any time soon, so this requires considerations.

Below is a copy paste of my comment in the discussion on my pull
request ( https://github.com/brettcannon/path-pep/pull/2 ). It is
about whether os.fspath should look like

def fspath(path, *, type_constraint=str): ...

This was already discussed before, receiving positive reactions. It
would be an extension to what is in the draft Brett just posted. Note
that the return type, however, does not depend on `type_constraint`.
It fully depends on `path`. The constraint only determines when an
exception is raised instead of returning a value.  When using
str-based paths normally, one would do

os.fspath(path)

Calling it with

os.fspath(path, type_constraint=(str, bytes))

would turn off rejection of bytes paths (and be consistent with the
rest of `os` and `os.path` where functions accept both types). But the
default value for the keyword argument would be a good reminder that
str-based paths should be used by default. This str-constraint is also
present in the current drafted version, but it is not optional or
visible in the signature.

(See the diffs of my pull request for an example implementation of this.)

So below is the copy-paste from the pull request discussion that I promised:

"""
Nobody wants to promote using `bytes` for paths without a proper
reason. However, `os.fspath(path, type_contraint=str)` is already a
compromise from the current `os.*` convention (of symmetrically
supporting both `str` and `bytes`) towards enforcing `str`-based
paths.

As you know, the reason for using `os.fspath` is to switch to a
lower-level representation of paths as strings. When you use it, you
are already deciding that you are lower-level and want to examine or
manipulate path strings manually. I think the `type_constraint=str`
keyword-only argument (with the default) is a good way to remind the
'user' that `str` is the way to go unless you know what you are doing
and, to `bytes` users, that `os.fspath` (by default) rejects bytes.

Looking at the discussions on python-dev, one notices that the active
people in the discussions were mostly in favor of exposing the
bytes-supporting version in one way or the other. Some were even
against bytes-rejecting versions.

Most people (Python programmers), especially in the long term, should
not be using `os.fspath` a lot. If they want to print a path, they
should simply do so: `print(path_obj)`, or `print(f"The path is:
{path_obj}")`. However, there will always be people that for whatever
reason want to convert a pathlib object into a string, and without
even consulting the docs, they may write `str(path_obj)`. And that's
not the end of the world. It's not any worse than `str(integer)` for
instance. After all, if the main point is to convert something into a
generic string, the function should probably be called `str`, not
`fspath`. But if the idea is to convert any path object into something
that `os` and similar code internally understand, then `os.fspath`
sounds fine to me. So I would not go as far as 'never use
`str(path)`'. What we don't want is things like `open(str(path),
'w')`. I know you agree, because you wrote that yourself. But once
this protocol is available, I see no reason why people would
voluntarily wrap their path objects in `str(...)`, so I don't see that
as a problem in the future (thanks to this PEP). So, especially in the
long term, I don't expect `fspath` to be an everyday tool for people.
Even in shorter term, people don't usually need it, because the stdlib
will already support pathlib objects.

Since it is not an everyday tool, I think it's ok to have that extra
keyword-only argument. Even if it were an everyday tool, I can see the
keyword-only argument in the signature as useful thing in a
documentation sense.

[As a side note, I could imagine someone convincing me that `fspath`
should *always* accept both `str`- and `bytes`-based paths, because
that would simplify it and align it with the rest of `os`. This would
not be any worse than the status quo in terms of early failures. That
would, however, still not detect accidental use of bytes paths (when
forgotten to decode from the encoding it was received in).]
"""

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 12:15 AM, Ethan Furman  wrote:
> On 05/11/2016 01:51 PM, Ethan Furman wrote:
>>
>> On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
>
>
 os.path
 '''

 The various path-manipulation functions of ``os.path`` [#os-path]_
 will be updated to accept path objects. For polymorphic functions that
 accept both bytes and strings, they will be updated to simply use
 code very much similar to
 ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
 will allow for their pre-existing type-checking code to continue to
 function.
>>>
>>>
>>> I afraid that this will hit a performance. Some os.path functions are
>>> used in tight loops, they are hard optimized, and adding support of path
>>> protocol can have visible negative effect.
>>
>>
>> Do you have an example of os.path functions being used in a tight loop?
>

I'd be interested in this too.

>
> Also, the C code for fspath can check types first and take the fast path if
> bytes/str are passed in, only falling back to the __fspath__ protocol if
> something else was passed in -- which should make any performance hits
> negligible.

My suggestion for the *python version* already does this too (again,
see my PR). This is a win-win, as it also improves error messages (as
suggested by Nick in the earlier discussions).

-- Koos

>
>
> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Wed, May 11, 2016 at 11:04 PM, Brett Cannon  wrote:
> A quick comment about sending me fixes. While I do appreciate them, sending
> them as a pull request is much easier for me as (a) I don't have to hunt the
> changes down in the text, and (b) you will see the fixes others have done
> already to the PEP and I then don't have to figure out what changes have not
> already been fixed. And honestly, reading the PEP in its rendered format on
> GitHub is easier IMO than in the text format unless you have something
> specific to respond to (and even if you do, you can copy and paste the
> relevant bits into an email reply).
>

Personally, I find it more important to settle the open issues first,
so I will be looking for typos at a later stage. Besides, more typos
may be added in the process. I previously left some typos uncorrected
in attempt to keep the diffs of my commits cleaner. By the way, to see
what additions and wordings I have suggested, the right way is to look
at my PR(s).  I might send more PRs to clarify open issues etc., but
this will largely be based on my previous commits.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 12:28 AM, Nikolaus Rath  wrote:
> On May 11 2016, Brett Cannon  wrote:
>> This PEP proposes a protocol for classes which represent a file system
>> path to be able to provide a ``str`` or ``bytes`` representation.
> [...]
>
> As I said before, to me this seems like a lot of effort for a very
> specific use-case. So let me put forward two hypothetical scenarios to
> better understand your position:

I think you are touching important points.

> - A new module for URL handling is added to the standard library (or
>   urllib is suitably extended). There is a proposal to add a new
>   protocol that allows classes to provide a ``str`` or ``bytes``
>   representation of URLs.

This reminds me of the thread I recently started on python-ideas about
extending the concept of paths to URLs. I don't know if you are
referring to something like that or not.

Anyway, it would be important to know whether the str or bytes
representation is to be used a file system path or an URL, so that
would need to be a separate protocol. But everyone would have the
experience from these discussions, so hopefully less discussion then
:). (By the way, this is one reason to have 'fs' in the name of the
__fspath__ method, although I have not mentioned it before.)

>
> - A new (third-party) library for natural language processing arises
>   that exposes a specific class for representing audio data. Existing
>   language processing code just uses bytes objects. To ease transition
>   and interoperability, it is proposed to add a new protocol for classes
>   that represend audio data to provide a bytes representation.
>
> Do you think you would you be in favor of adding these protocols to
> the stdlib/languange reference as well? If not, what's the crucial
> difference to file system paths?
>

File system paths are very fundamental, and will probably be used in
context of your natural language example too.

-- Koos

>
> Thanks,
> -Nikolaus
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 3:04 PM, Nick Coghlan  wrote:
>
> I'd still like to see this exposed to Python code as os._raw_fspath()
> (with the leading underscore just meaning "this probably isn't the API
> you want" rather than indicating a private or unstable API), and then
> fspath() defined as a wrapper around it which disallows bytes as
> output.
>

I don't remember (should probably check) if you previously proposed
implementing exactly that in C, but I indeed agree with what you write
above, except that I don't like the "_raw_" prefix in the name. I
would be willing to call that simply fspath though, since as mentioned
before in this thread (I think by Brett and me), the reasons for
rejecting bytes in fspath are really quite minor.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 4:20 PM, Nick Coghlan  wrote:
>
> It's not unusual for me to encounter "POSIX oughtta be enough for
> anyone" folks that are not yet entirely convinced that
> bytes-are-not-text, so I'm actually in favour of making the default
> Python-level API str-only as a healthy nudge away from the
> "text-is-just-bytes-with-an-encoding!" school of thought.
>

This was also how I convinced myself about the default str constraint.
However, I'm afraid it would be a weak weapon against using bytes
paths, since the people using bytes paths would not be likely to call
it, regardless of whether it supports bytes or not.

The nice thing is that pathlib is str-only and *that* will push people
away from bytes paths.

> However, in terms of the three groups Brett articulated (maximum
> flexibility, encouraging cross-platform correctness, and forgetting
> the whole idea), I'm in both camps 1 & 2 - I work with POSIX enough
> that I'm entirely on board with the notion that if you're specifically
> modelling *POSIX* paths, then bytes-with-an-assumed-encoding is
> frequently a good enough representation, but also deal with other
> environments (like Windows, the JVM and the CLR) enough to know that
> that particular representation of filesystem paths breaks down the
> moment you expand your scope of interest beyond *nix platforms.
>

I also agree with parts about Brett's "camp 2".

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 2:53 AM, Koos Zevenhoven <k7ho...@gmail.com> wrote:
> On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>> On Thu, May 12, 2016 at 2:05 AM, Ethan Furman <et...@stoneleaf.us> wrote:
>>> On 05/11/2016 03:13 PM, Brett Cannon wrote:
>>>
>>>> If [...] I would drop os.path changes and make os.fspath() do what
>>>
>>>> Ethan and Koos have suggested and simply pass through without checks
>>>>
>>>> whatever path.__fspath__() returned if the argument wasn't str or bytes.
>>>
>>>
>>> Not to derail the conversation too much, as I know we're all getting burned
>>> out on the topic, but that last bit is not accurate: my druthers are to have
>>> __fspath__ be able to return str /or/ bytes, and if anything else comes from
>>> the object in question an exception must be raised. Maybe a word got lost
>>> between your thoughts and your fingers -- happens to me all the time.  :)
>>
>> Yes. This would also be equivalent to my fspath(path,
>> type_constraint=(str,bytes)). And if the compromise I mentioned about
>> the rejecting (by default or optionally) is lifted, the keyword
>
> the rejecting -> rejecting bytes
>
>> argument would not be needed.  I might be ok with throwing away the
>> isinstance check on the return value of __fspath__() if it has
>> significant impact on performance in realistic cases (with DirEntry
>> most likely, I suppose), but I doubt it.
>>
>> -- Koos
>>

I will send a pull request about this tomorrow.

-- Koos

>>>
>>> --
>>> ~Ethan~
>>>
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 2:05 AM, Ethan Furman  wrote:
> On 05/11/2016 03:13 PM, Brett Cannon wrote:
>
>> If [...] I would drop os.path changes and make os.fspath() do what
>
>> Ethan and Koos have suggested and simply pass through without checks
>>
>> whatever path.__fspath__() returned if the argument wasn't str or bytes.
>
>
> Not to derail the conversation too much, as I know we're all getting burned
> out on the topic, but that last bit is not accurate: my druthers are to have
> __fspath__ be able to return str /or/ bytes, and if anything else comes from
> the object in question an exception must be raised. Maybe a word got lost
> between your thoughts and your fingers -- happens to me all the time.  :)

Yes. This would also be equivalent to my fspath(path,
type_constraint=(str,bytes)). And if the compromise I mentioned about
the rejecting (by default or optionally) is lifted, the keyword
argument would not be needed.  I might be ok with throwing away the
isinstance check on the return value of __fspath__() if it has
significant impact on performance in realistic cases (with DirEntry
most likely, I suppose), but I doubt it.

-- Koos

>
> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 1:13 AM, Brett Cannon  wrote:
>
>
> On Wed, 11 May 2016 at 13:45 Serhiy Storchaka  wrote:
>>
>> On 11.05.16 19:43, Brett Cannon wrote:
>> > os.path
>> > '''
>> >
>> > The various path-manipulation functions of ``os.path`` [#os-path]_
>> > will be updated to accept path objects. For polymorphic functions that
>> > accept both bytes and strings, they will be updated to simply use
>> > code very much similar to
>> > ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
>> > will allow for their pre-existing type-checking code to continue to
>> > function.
>>
>> I afraid that this will hit a performance. Some os.path functions are
>> used in tight loops, they are hard optimized, and adding support of path
>> protocol can have visible negative effect.
>
>
> As others have asked, what specific examples do you have that os.path is
> used in a tight loop w/o any I/O that would overwhelm the performance?
>
>>
>>
>> I suggest first implement other changes and then look whether it is
>> worth to add support of path protocol in os.path functions.
>
>
> I see this whole discussion breaking down into a few groups which changes
> what gets done upfront and what might be done farther down the line:
>
> 1. Maximum acceptance: do whatever we can to make all representation of paths
> just work, which means making all places working with a path in the stdlib
> accept path objects, str, and bytes.

Since you are putting me in this camp, there is at least one thing you
are wrong about. I don't want all places that work with a path to
accept bytes. Only those that already do so, including os/os.path. And
yes, I think the stdlib should show a good example in accepting path
types (especially those provided in the stdlib itself).

Whether Ethan is fully in camp 1, I don't know. Not that I think he
would be any closer to the other camps, though.

> 2. Safely use path objects: __fspath__() is there to signal an object is a 
> file
> system path and to get back a lower-level representation so people stop
> calling str() on everything, providing some interface signaling that someone
> doesn't misuse an object as a path and only changing path consumptions APIs
> -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the
> stdlib.
>
> 3. It ain't worth it: those that would rather just skip all of this and drop
> pathlib from the stdlib.
>
> Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to
> compromise somewhat and find a middle ground in the PEP with the level of
> changes in the stdlib but being more restrictive with os.fspath(). If I were
> doing a pure group #2 PEP I would drop os.path changes and make os.fspath()
> do what Ethan and Koos have suggested and simply pass through without checks
> whatever path.__fspath__() returned if the argument wasn't str or bytes.
>

Related to this, based on the earlier discussions, I had the
impression that you were largely in the same camp as me. In fact, I
thought you had politely left some things out of the PEP draft so I
could fill them in. It turned out I was wrong about that, because you
didn't merge them.

-- Koos

> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven <k7ho...@gmail.com> wrote:
> On Thu, May 12, 2016 at 2:05 AM, Ethan Furman <et...@stoneleaf.us> wrote:
>> On 05/11/2016 03:13 PM, Brett Cannon wrote:
>>
>>> If [...] I would drop os.path changes and make os.fspath() do what
>>
>>> Ethan and Koos have suggested and simply pass through without checks
>>>
>>> whatever path.__fspath__() returned if the argument wasn't str or bytes.
>>
>>
>> Not to derail the conversation too much, as I know we're all getting burned
>> out on the topic, but that last bit is not accurate: my druthers are to have
>> __fspath__ be able to return str /or/ bytes, and if anything else comes from
>> the object in question an exception must be raised. Maybe a word got lost
>> between your thoughts and your fingers -- happens to me all the time.  :)
>
> Yes. This would also be equivalent to my fspath(path,
> type_constraint=(str,bytes)). And if the compromise I mentioned about
> the rejecting (by default or optionally) is lifted, the keyword

the rejecting -> rejecting bytes

Good night.

-- Koos


> argument would not be needed.  I might be ok with throwing away the
> isinstance check on the return value of __fspath__() if it has
> significant impact on performance in realistic cases (with DirEntry
> most likely, I suppose), but I doubt it.
>
> -- Koos
>
>>
>> --
>> ~Ethan~
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 7:24 PM, Guido van Rossum  wrote:
> I am glad this is finally happening. There's quite a bit of noise in the
> thread which I have to ignore. The two issues that I want to respond to are
> speed and whether os.fspath() can return bytes.
>
> - Speed: We should trust our ability to optimize the implementations where
> necessary. First the API issues need to be settled.
>
> - Bytes: I strongly believe that os.fspath() should be a thin wrapper around
> the __fspath__ protocol, like next() wraps the .__next__ protocol. It should
> not get into bytes vs. string politics. If your app really needs strings,
> call os.fsdecode(). So this is my version (unoptimized):
>

:)

Thank you for this. I can breathe now.

Some questions remain:

> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
> if isinstance(p, (str, bytes)):
> return p
> try:
> return p.__fspath__
> except AttributeError:
> raise TypeError(...)
>

(I know Brett already posted this question, but somehow it did not
show up in my mailbox before I had written this. I'm (re)posting
because there is some stuff here that is not in Brett's email )

You might be suggesting that __fspath__ should be an attribute, not a
method, or did you mean something like:

def fspath(p):
if isinstance(p, (str, bytes)):
return p
try:
p.__fspath__
except AttributeError:
raise TypeError(...)
return p.__fspath__()

IMO, either is fine, I suppose. As you know, it's mostly a question of
whether __fspath__ will be a property or a method (on PurePath for
instance). But if you meant the former, that would change also the ABC
and the protocol description.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 8:22 PM, Sjoerd Job Postmus
 wrote:
> I would like to make just 1 comment regarding the question of accepting
> (or not) bytes as output of `os.fspath`.
>
> The whole point of adding `os.fspath` is to make it easier to use Path
> objects. This is in an effort to gain greater adoption of pathlib in
> libraries. Now, this is an excellent idea.
>
> However, if it were to reject bytes, that would mean that when libraries
> start to use pathlib, it would suddenly become harder for people that
> actually need bytes-support to use pathlib.
>
> Now, the claim 'if you need bytes, you should not be using pathlib` is a
> reasonable one. But what if I need bytes *and* a specific library (say,
> image handling, or a web framework, or ...). It's not up to me if that
> library uses pathlib or plain old os.path.join.
>
> Is using surrogate-escapes enough for this case? I myself am not sure,
> (and also not affected), but it sounds to me that rejecting bytes is a
> wrong approach if there is no proper workaround (assuming the use-case
> of pathlib is somewhere deep in library code).
>

This is out of the scope of this PEP and probably a very insignificant
issue (luckily, this is not the pathlib PEP). Surrogates will probably
work and if not, on can "blaim" broken filenames ;).

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

This has just been discussed very recently in this thread (and earlier
too). It may make sense, but it's not among our current worries.
Besides, we already added the new fspath semantics to the PEP.

While I hope Brett is asleep in his time zone, I'm guessing he will
agree (just saying this because you write "@Brett").

-- Koos


On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze  wrote:
> On 12.05.2016 18:24, Guido van Rossum wrote:
>>
>> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
>> if isinstance(p, (str, bytes)):
>> return p
>> try:
>> return p.__fspath__
>> except AttributeError:
>> raise TypeError(...)
>
>
> @Brett
> Would you think it makes sense to swap the str/bytes check and the
> __fspath__ check?
>
>
> I just thought of a class subclassing str/bytes and defines __fspath__. Its
> __fspath__ method would be ignored currently.
>
>
> Best,
> Sven
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Fri, May 13, 2016 at 12:24 PM, Sven R. Kunze <srku...@mail.de> wrote:
> On 13.05.2016 10:36, Koos Zevenhoven wrote:
>>
>> This has just been discussed very recently in this thread (and earlier
>> too).
>
>
> Could you point me to that? It seems I missed that part. I only found posts
> related to performance degradation.
>

This issue is coupled with the future optimization questions.

> However, the proposed semantics will change if the checks are swapped. So,
> my actual question is:
>
> Is that an intended API inconsistency or a known bug supposed to be resolved
> later?
>

Taking into account the description (and the drafted type hint), which
the documentation will probably reflect, the semantic effects of that
are very minor or nonexistent.

I do think the documentation of the protocol should say that str or
bytes subclasses should not implement __fspath__.

So no API inconsistency there.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path PEP, part 2

[Koos Zevenhoven]

Thanks Brett!

Now one thing is that, despite your suggestion, I had not added myself
as an author in my big pull request. Originally, it was because I
simply forgot to copy and paste it when I split my edits into separate
commits ;-). Sorry about that (not sure if you care though, and I've
been defending the PEP regardless).

Anyway, especially now that my main worry regarding the open questions
has been resolved, I would be more than happy to have my name on it.
So Brett, could you add me as author? (Koos Zevenhoven and
k7ho...@gmail.com will be fine)

It looks like this is finally happening :)

-- Koos

On Thu, May 12, 2016 at 11:53 PM, Brett Cannon <br...@python.org> wrote:
> Second draft that takes Guido's comments into consideration. The biggest
> change is os.fspath() now returns whatever path.__fspath__() returns instead
> of restricting it to only str.
>
> Minor changes:
> - Renamed the C function to PyOS_FSPath()
> - Added an Implementation section with a TODO list
> - Bunch of things added to the Rejected Ideas section
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path PEP, part 2

[Koos Zevenhoven]

On Fri, May 13, 2016 at 2:00 PM, Steven D'Aprano  wrote:
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
>   from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;
>
> - there is no os function that returns "str or bytes, I don't
>   care which". (If you really need that, call __fspath__ directly.)
>
> Note that this differs from the already rejected suggestion that there
> should be two dunder methods, __fspath__() and __fspathb__().
>
> Why?
>
> (1) Normally, the caller knows whether they want str or bytes. (That's
> been my experience, you may disagree.) If so, and they call os.fspath()
> expecting a str, they won't be surprised by it returning bytes. And visa
> versa for when you expect a bytes path.
>
> (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.

I would think these have the b suffix because there is no good way to
infer which type should be returned. In things like os.path.join or
os.path.dirname you pass in the object(s) that determine the return
type.

In os.fspath, you pass in an object, whose type (str/bytes) or
"underlying path string type" (as returned by __fspath__()) determines
the return type of fspath. I think this is well in line with os.path
functions.


-- Koos

>
> Cons:
>
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Fri, May 13, 2016 at 7:06 PM, Sven R. Kunze <srku...@mail.de> wrote:
> On 13.05.2016 11:48, Koos Zevenhoven wrote:
>>
>> This issue is coupled with the future optimization questions.
>>
>
> AFAIC coupling API design to optimization is called premature optimization.
>

I suppose so, but currently I consider the API not to make guarantees
about what happens if you pass in something other than specified as
valid input. I'm not sure what level of precision is usual at this
point, but I think this is sufficient for now.

I'm sure we will pay attention to this (as we have been doing, both
while optimizing prematurely and for other reasons).

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path PEP, 3rd draft

[Koos Zevenhoven]

FYI, I recently sent a couple of emails in my earlier type hinting
thread on -ideas. What I wrote now is about the path ABC regarding
type hints.

-- Koos

On Sat, May 14, 2016 at 12:48 AM, Brett Cannon  wrote:
>
>
> On Fri, 13 May 2016 at 14:30 Philip Jenvey  wrote:
>>
>>
>> On May 13, 2016, at 11:37 AM, Brett Cannon  wrote:
>>
>> Biggest changes since the second draft:
>>
>> Resolve __fspath__() from the type, not the instance (for Guido)
>>
>>
>> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>> return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
>> path,
>> NULL);
>>
>>
>> _PyObject_LookupSpecial would be preferable then.
>
>
> Yes it would be. :) I'll add it to the PEP.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 11:31 AM, Sven R. Kunze  wrote:
> On 11.05.2016 18:43, Brett Cannon wrote:
>>
>> Rationale
>> =
>>
>> Historically in Python, file system paths have been represented as
>> strings or bytes. This choice of representation has stemmed from C's
>> own decision to represent file system paths as
>> ``const char *`` [#libc-open]_. While that is a totally serviceable
>> format to use for file system paths, it's not necessarily optimal. At
>> issue is the fact that while all file system paths can be represented
>> as strings or bytes, not all strings or bytes represent a file system
>> path.
>
>
> I can remember this argument being made during the discussion. I am not sure
> if that 100% correct as soon as we talk about PurePaths.
>

I had suggested an alternative wording for this (see my commit on the
work on Rationale).


>> Proposal
>> 
>>
>> This proposal is split into two parts. One part is the proposal of a
>> protocol for objects to declare and provide support for exposing a
>> file system path representation.
>
>
> https://docs.python.org/3/whatsnew/changelog.html says:
>
> "Add ‘path’ attribute to pathlib.Path objects, returning the same as str(),
> to make it more similar to DirEntry. Library code can now write getattr(p,
> ‘path’, p) to get the path as a string from a Path, a DirEntry, or a plain
> string. This is essentially a small one-off protocol."
>
> So, in order to promote the "small one-off protocol" to a more broader
> protocol, this PEP proposes a simple rename of .path to .__fspath__, is that
> correct?
>

Well, I have brought this up previously several times. Indeed I see
this as a further development of that duck-typing compatiblity
approach.  However, while the .path attribute is prior art, it has not
been in a release yet.

> Unfortunately, I don't have anything to contribute to the open issues. All
> solutions have their pros and cons and everything that could be said has
> been said. I think you need to decide.
>

Surprising enough, there are new things being said all the time. But
luckily there seem to be signs of convergence.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Thu, May 12, 2016 at 11:14 AM, Serhiy Storchaka  wrote:
>
> This is cheap in C, but os.path functions are implemented in Python. They
> have to make at least one function call (os.fspath(), hasattr() or
> isinstance()), not counting a bytecode for retrieving arguments, resolving
> attributes, comparing, jumps. Currently os.path functions use tricks to
> avoid overheads
>

I suppose a C-implemented version of fspath *called from python* might
be the fastest option at least in some cases. After all, a function
call (isinstance or hasattr) is likely anyway, unless of course `try:
path.__fspath__` is used.

> Yet one problem is that currently many os,path functions work with
> duck-typed strings (e.g. UserString). Using os.fspath() likely limit
> supported types to str, bytes and types that support the path protocol.
>

Something like

path = path.__fspath__() if hasattr(path, '__fspath__') else path

as currently in the PEP, would not have this problem. However, I
wonder whether such duck string paths actually exist (although it does
remind me of my earlier experiments for solving the pathlib
compatibility problem ;-).


-- Koos

P.S: I think it's great that you are concerned about the performance,
and I find it important. However, this does feel like premature
optimization to me at this point. We should first decide what
functions should support the protocol, and after that, find the
performance concerns and fix them. I have the feeling that the cases
where this would be a performance bottleneck are quite rare, but if
they are not, they may well be fixable.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] File system path PEP, part 2

[Koos Zevenhoven]

On Fri, May 13, 2016 at 8:52 PM, Steven D'Aprano  wrote:
> On Fri, May 13, 2016 at 03:43:29PM +, Brett Cannon wrote:
>> On Fri, 13 May 2016 at 04:00 Steven D'Aprano  wrote:
>
> [...]
>> > I think it is a bit confusing to refer to "path objects", as that seems
>> > like you are referring only to pathlib.Path objects. It took me far too
>> > long to realise that here you mean generic path-like objects that obey
>> > the __fspath__ protocol rather than a specific concrete class.
>> >

This terminology is indeed a bit difficult, not least because there
are 6 different path classes in pathlib. A couple of months ago, I
decided to start to call these pathlib objects and path objects,
because I did not know what else to call them without doubling the
length.

>> > Since the ABC is called "PathLike", I suggest we refer to "path-like
>> > objects" rather than "path objects", both in the PEP and in the Python
>> > docs for this protocol.
>>
>> I went back and forth with this in my head while writing the PEP. The
>> problem with making "path-like" mean "objects implementing the PathLike
>> ABC" becomes how do you refer to an argument of a function that accepts
>> anything os.fspath() does (i.e. PathLike, str, and bytes)?
>
> On further reflection, I think the right language is to use "path-like"
> for the union of Pathlike, str and bytes. That will, I think, cover the
> majority of cases: most functions which want to work on a file system
> path should accept all three. When you want to specify only an object
> which implements the PathLike ABC, that's called a (virtual) instance of
> PathLike.
>

As I've told Brett before, I think exactly this reasoning would be a
good enough reason not to call the ABC PathLike.

[...]
>> > Would I be right in saying that in practice this will actually end up
>> > being type(path).__fspath__() to match the behaviour of all(?) other
>> > dunder methods?
>>
>> I wasn't planning on it because for most types the accessing of the method
>> directly off of the type for magic methods is because of some special
>> struct field at the C level that we're pulling from. Since we're not
>> planning to have an equivalent struct field I don't see any need to do the
>> extra work of avoiding the instance participating in method lookup.
>> Obviously if people disagree for some reason then please let me know (maybe
>> for perf by avoiding the overhead of checking for the method on the
>> instance?).
>
> The reasons I would disagree are:
>
> (1) It took me a long time to learn the rule that dunder methods are
> always called from the class, not the instance, and now I have to learn
> that there are exceptions? G argggh.

Who knows, maybe the exception made it take a longer time..

IIRC the docs say that dunder methods are *not guaranteed* to be
called on the instance, because they may be called on the class.

> (2) If we ever do change to a C struct field, the behaviour will change.
> Maybe it's better to emulate the same behaviour from the start?
>
> (3) If there's a performance speed up, that's a bonus!
>
>
>
> --
> Steven
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Koos Zevenhoven]

On Fri, May 13, 2016 at 7:43 PM, Chris Angelico  wrote:
[...]
> "Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x)
> is str"). The question is, which one SHOULD be being done?

Indeed it should do "Check", so that path libraries that do inherit
from str will still work (see also below).

> What should this do:
>
> class TmpPath(str):
> def __fspath__(self):
> return "/tmp/"+self
> x = TmpPath("foo/bar")
> open(x, "w")
>
> Does that respect __fspath__, or respect the fact that it's a string?

It should not respect __fspath__ because that would mean the behavior
is different in older versions of Python(!). This also means that such
code should never be written (except for that example you wrote ;-).
So indeed, composition would be the way to go.

So yes, I still think str subclasses should not implement __fspath__,
and that, at the very least, the docs should say that :).

-- Koos


> ChrisA
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Koos Zevenhoven]

On Thu, Apr 14, 2016 at 9:35 PM, Random832 <random...@fastmail.com> wrote:
> On Thu, Apr 14, 2016, at 13:56, Koos Zevenhoven wrote:
>> (1) Code that has access to pathname/filename data and has some level
>> of control over what data type comes in. This code may for instance
>> choose to deal with either bytes or str
>>
>> (2) Code that takes the path or file name that it happens to get and
>> does something with it. This type of code can be divided into
>> subgroups as follows:
>>
>>   (2a) Code that accepts only one type of paths (e.g. str, bytes or
>> pathlib) and fails if it gets something else.
>
> Ideally, these should go away.
>

I don't think so. (1) might even be the most common type of all code.
This is code that gets a path from user input, from a config file,
from a database etc. and then does things with it, typically including
passing it to type (2) code and potentially getting a path back from
there too.

>>   (2b) Code that wants to support different types of paths such as
>> str, bytes or pathlib objects. This includes os.path.*, os.scandir,
>> and various other standard library code. Presumably there is also
>> third-party code that does the same. These functions may want to
>> preserve the str-ness or bytes-ness of the paths in case they return
>> paths, as the stdlib now does. But new code may even want to return
>> pathlib objects when they get such objects as inputs.
>
> Hold on. None of the discussion I've seen has included any way to
> specify how to construct a new object representing a different path
> other than the ones passed in. Surely you're not suggesting type(a)(b).
>

That's right. This protocol is not solving the issue of returning
'rich' path objects. It's solving the issue of passing those objects
to lower-level functions or to interact with other 'rich' path types.
What I meant by this is that there may be code that *does* want to do
type(a)(b), which is out of our control. Maybe I should not have
mentioned that.

> Also, how does DirEntry fit in with any of this?
>

os.scandir + DirEntry are one of the many things in the stdlib that
give you pathnames of the same type as those that were put in.

>> This is the
>> duck-typing or polymorphic code we have been talking about. Code of
>> this type (2b) may want to avoid implicit conversions because it makes
>> the life of code of the other types more difficult.
>
> As long as the type it returns is still a path/bytes/str (and therefore
> can be accepted when the caller passes it somewhere else) what's the
> problem?

No, because not all paths are passed to the function that does the
implicit conversion, and then when for instance os.path.joining two
paths of a differenty type, it raises an error.

In other words: Most non-library code (even library code?) deals with
one specific type and does not want implicit conversions to other
types. Some code (2b) deals with several types and, at least in the
stdlib, such code returns paths of the same type as they are given,
which makes said "most non-library code" happy, because it does not
force the programmer to think about type conversions.

(Then there is also code that explicitly deals with type conversions,
such as os.fsencode and os.fsdecode.)

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Koos Zevenhoven]

On Tue, Apr 19, 2016 at 2:55 PM, Stephen J. Turnbull  wrote:
>
> AFAICS bytes return from __fspath__ is just YAGNI.  Show me something
> that actually wants it.

It might be, but as long as bytes paths are supported polymorphicly
all over the stdlib, we won't get rid of supporting bytes paths. So
are you proposing to deprecate bytes paths?

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Koos Zevenhoven]

On Sun, Apr 17, 2016 at 11:03 AM, Stephen J. Turnbull
 wrote:
> Nick Coghlan writes:
>
>  > str and bytes aren't going to implement __fspath__ (since they're
>  > only *sometimes* path objects), so asking people to call the
>  > protocol method directly for any purpose would be a pain.
>
> It *should* be a pain.  People who need bytes should call fsencode,
> people who need str should call fsdecode, and Ethan's antipathy checks
> for bytes and str, then calls __fspath__ if needed.  Who's left?  Just
> the bartender and the janitor, last call was hours ago.  OK, maybe
> there are enough clients to make it worthwhile to provide the utility,
> but it should be clearly marked as "double opt-in, for experts only
> (consenting adults must show proof of insurance)".

My doubts, expressed several times in these threads, about the need
for a *public* os.fspath function to complement the __fspath__
protocol, are now perhaps gone. I'll explain why (and how). The
reasons for my doubts were that

(1) The audience outside the stdlib for such a function should be
small, because it is preferred to either use existing tools in
os.path.* or pathlib (or similar) for manipulating paths.

(2) There are just too many different possible versions of this
function: rejecting str, rejecting bytes, coercion to str, coercion to
bytes, and accepting both str and bytes. That's a total of 5 different
cases. People also used to talk about versions that would not allow
passing through objects that are already bytes or str. That would make
it a total of 10 different versions!
(in principle, there could be even more, but let's not go there :-).
In other words, this argument was that it is probably best to
implement whatever flavor is needed for the context, perhaps based on
documented recipes.


Regarding (2), we can first rule out half of the 10 cases---the ones
that reject plain instances of bytes and/or str---because they would
not be very useful as all the isinstance/hasattr checking etc. would
be left to the caller. And here are the remaining five, explained
based on what they accept as argument, what they return, and where
they would be used:

(A) "polymorphic"
*Accept*: str and bytes, provided via __fspath__ as well as plain str
and bytes instances.
*Return*: str/bytes depending on input.
*Audience*: the stdlib, including os.path.things, os.things,
shutil.things, open, ... (some functions would need a C version).
There may even be a small audience outside the stdlib.

(B) "str-based only"
*Accept*: str, provided via __fspath__ as well as plain str.
*Return*: str.
*Audience*: relatively low-level code that works exclusively with str
paths but accepts specialized path objects as input.

(C) "bytes-based only"
*Accept*: bytes, provided via __fspath__ as well as plain bytes.
*Return*: bytes.
*Audience*: low-level code that explicitly deals with paths as bytes
(probably to deal with undefined/ill-defined encodings).

(D) "coerce to str"
*Accept*: str and bytes, provided via __fspath__ as well as plain str
and bytes instances.
*Return*: str (coerced / decoded if needed).
*Audience*: code that deals explicitly with str but wants to 'try'
supporting bytes-based path inputs too via implicit decoding (even if
it may result in surrogate escapes, which one cannot for instance
print(...).)

(E) "coerce to bytes"
*Accept*: str and bytes, provided via __fspath__ as well as plain str
and bytes instances.
*Return*: bytes (coerced / encoded if needed).
*Audience*: low-level code that explicitly deals with bytes paths but
wants to accept str-based path inputs too via implicit encoding.


Even if all options (A-E) probably have small audiences (compared to
e.g. os.path.*), some of them have larger audiences than others. But
all of them have at least *some* reasonable audience (as desribed
above).

Recently (well, a few days ago, but 'recently', considering the scale
of these discussions anyway ;-), Nick pointed out something I hadn't
realized---os.fsencode and os.fsdecode actually already implement
coercion to bytes and str, respectively. With those two functions made
compatible with the __fspath__ protocol [using (A) above], they would
in fact *be* (D) and (E), respectively.

Now, we only have options (A-C) left. They could all be implemented
roughly as follows:

def fspath(pathlike, *, output_types = (str,)):
  if hasattr(pathlike, '__fspath__'):
ret = pathlike.__fspath__()  # or pathlike.__fspath__ if it's not a method
  else:
ret = pathlike
  if not isinstance(ret, output_types):
raise TypeError("argument is not and does not provide an
acceptable pathname")
  return ret

With an implementation like the above, (A) would correspond to
output_types = (str, bytes), (B) to the default, and (C) to
output_types = (bytes,).


So, with the above considerations as a counterargument, I consider
argument (2) gone.

What about argument (1), that the audience for the os.fspath(...)
function (especially for one selected version of 

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Koos Zevenhoven]

On Sun, Apr 17, 2016 at 9:14 PM, Ethan Furman <et...@stoneleaf.us> wrote:
> On 04/17/2016 06:58 AM, Koos Zevenhoven wrote:
>
>> So, as a summary: With a str+bytes-polymorphic __fspath__, with the
>> above argumentation and the rough implementation of os.fspath(...),
>> the conclusion is that the os.fspath function should indeed be public,
>> and that no further variations are needed.
>
>
> Nice summation, thank you.  :)
>

Come on, Ethan, that summary was not for you ;) It was for lazy
people, people with bad memory, or people not so involved in the
topic. I wrote a big post, provided new arguments, with other points
collected into the same logical framework, wrote a new version of
os.fspath and argued why it is the right one --- and all you do is
read the stupid summary. You can do better than that: read the whole
thing! ;-).

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Koos Zevenhoven]

On Wed, Apr 20, 2016 at 6:11 AM, Stephen J. Turnbull <step...@xemacs.org> wrote:
> Koos Zevenhoven writes:
>  > On Tue, Apr 19, 2016 at 2:55 PM, Stephen J. Turnbull <step...@xemacs.org> 
> wrote:
>  > >
>  > > AFAICS bytes return from __fspath__ is just YAGNI.  Show me something
>  > > that actually wants it.
>  >
>  > It might be,
>
> May I take that as meaning you just jumped to the conclusion that
> extending polymorphism is useful on no actual evidence of usefulness?

No you may not! YAGNI almost never means "you are *never* going to
need it". And if you implement a feature, better implement it well. If
a variation of the feature is rarely used, that is perfectly fine. I
think leaving bytes out would complicate things. If os.fspath does its
job well, everyone should be happy.

I kept bringing up bytes paths, because that is already a feature in
Python 3. Then (already some time ago in these discussions) I briefly
visited the thought of 'can we deprecate bytes paths', and it then
quickly became clear to me that is not going to happen any time soon.

In other words: As long as bytes paths are supported, they should be
supported consistently. I don't want DirEntry to behave differently
when the underlying type is bytes, which is one of the things I've
been talking about all the time. That would just be broken. And as you
also understand, one point is to allow passing DirEntry to open. Or
any of the os.path functions.

An some more: I don't want open(direntry_obj) to ever raise because it
is the bytes flavor of direntry, because, when they are created,
DirEntry objects always point to existing objects on the file system.
I also don't want implicit conversions between str and bytes paths,
because there are cases where they will produce strange results and
exceptions. [Yes, way back in the p-string thread, I did first suggest
a similiar thing that implied implicit conversion, but I soon
abandoned that part.]

Not that I will ever use these features---just to do this right.

>  > but as long as bytes paths are supported polymorphicly all over the
>  > stdlib, we won't get rid of supporting bytes paths. So are you
>  > proposing to deprecate bytes paths?
>
> You claim "almost always want str", Ethan claims "bias against bytes."
> Sorry, guys, you can't have it both ways.  Either bytes paths are
> discouraged (not "deprecated", not yet), or they aren't.
>
> I say, let's not encourage them.

It's all essentially the same thing:

"almost always want str":
Yes, I still claim this. This is the reason for str (and rejecting
bytes) being the default for third-party code. If we wanted to, we
could even leave bytes support out of the documentation, so no-one
will know about it unless they already deal with bytes paths. However,
I dont think we should do that---we should just strongly discourage
using the bytes version unless there is a reason to, and you know what
you are doing.

"bias against bytes":
I agree with this too. This is in line with making str (and rejecting
bytes) the default for third-party code.

"let's not encourage them":
And I even agree with this, as you may have noticed.

I just don't believe in deliberately making implementations awkward
for the bytes-based paths. Bytes paths already exist, not because of
Python 2 (as you know), but because not all operating systems
guarantee that paths make sense in any encoding, and people may need
to work at that level.

There is no need to make working with bytes-based paths awkward, and
we can support them with little additional work compared to supporting
str-based rich path objects. The additional work is mostly this
discussion.

> Ie, keep the status quo for bytes,
> and make things better for the preferred str.  Yes, that means
> discouraging bytes relative to str in this context.  That's a Python 3
> principle, one strong enough to justify the huge compatibility break
> involved in making str be Unicode.  That compatibility break has been
> extremely successful in my personal experience as a sometime Python
> teacher and Mailman developer, though the Mercurial developers have a
> different POV.

Yes. Luckily, people are already using str-based paths. We don't need
any more discrete transitions. If linux will start to enforce an
encoding, as Guido and Random832 may be suggesting on python-ideas,
these already obscure bytes paths will slowly fade away.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Koos Zevenhoven]

On Wed, Apr 20, 2016 at 6:16 AM, Stephen J. Turnbull  wrote:
>
> (1) some really attractive producer of pathlib.Paths will be
> published, and
>

Yes, pathlib is str-only, so this sounds just right.

> (2) people will want to plug that producer into their bytes paths
> consumers using os.fspath(path) "and be done with it".
>

No, fspath can't know that is the the right thing to do.  There should
be *someone* that is aware of the encoding that happens, either the
provider or the consumer. That byte path consumer, assuming it wants
to support the behavior you describe, should use os.fsencode instead
of os.fspath, which will do exactly what you want, and just as easy
for the bytes path consumer to implement!

(Unless you want to explicitly reject plain str objects, which you
would then indeed do *explicitly*, but I'm not sure there is a point
in accepting plain bytes and str-based pathlib objects but not str).

To avoid further unnecessary discussion, please read [1] carefully,
where I already explained this, among other things.

-Koos

[1] https://mail.python.org/pipermail/python-dev/2016-April/144239.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion on fspath: please wait for a PEP

[Koos Zevenhoven]

On Wed, Apr 20, 2016 at 2:52 PM, Victor Stinner
 wrote:
> Hi,
>
> I'm unable to count the number of threads about the fspath protocol.
> It's even more difficult to count the total number of emails. IMHO
> everyone had enough time to give him/her opinion.

Couldn't agree more.

> We even had multiple
> summaries :-)

I'm not quite as sure about this. Maybe the meaning of "summary" in
the subculture of python lists is different from the one I know.

> Can you please wait for a PEP? Brett Canon and Ethan Furman are
> working on a PEP. So please give them time to write it.

I wonder what happened there...

> The PEP should summarize the discussion and help a lot to make
> concrete progress on the design (avoid restarting to discuss the same
> points forever). I don't expect that more emails would add anything at
> the current state of the discussion.

Again, agreed, and this part makes me feel relieved. Personally, I got
tired of the discussion a long time ago, but felt it had to be
finished.

> I think that we have enough other topics to discuss in the meanwhile ;-)

No doubt about that.

> FYI there is already an article about fspath/pathlib on LWN. Here is a
> free link until the article is freely accessible:
>
> "Python looks at paths" By Jake Edge (April 13, 2016)
> https://lwn.net/SubscriberLink/683350/4f52334af09653c8/

Wow. Wasn't expecting that. A whole story about the notorious "path
discussions"! (well, up to some date). Anyway, the beginning seems
fairly accurate, but then, among other things, it fails to mention
this for example:

https://mail.python.org/pipermail/python-ideas/2016-March/039179.html
https://mail.python.org/pipermail/python-ideas/2016-April/039595.html

Since I did not get any responses to that suggestion, it felt like a
dead end, and I continued experimenting with other things and ended up
taking the approach of "subclassing path-types from str gives more
complete pathlib support, but the objects should not pretend to be
strings in every way". By the way, I even implemented this, which I
suppose I failed to mention. Admittedly, it became a little awkward in
the end, but the main point was to provide a smooth transition from a
str world to a PurePath-subclass world (as opposed to a discrete one
like Py3k).

While I was working on that, the discussions on -dev seemed to have
reopened the gate at exactly that 'dead end' I mentioned before, and
had started to step through it.

-Koos

>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - improve fsdecode and fsencode

[Koos Zevenhoven]

On Thu, Apr 14, 2016 at 9:55 AM, Stephen J. Turnbull  wrote:
> Please please please, junk both "filter out bytes" proposals.

If you were referring to some of the fspath versions, I think we will
need a bytes-rejecting version, for reasons explained in [1-2]. Of
course not eveŕyone wants or has to use it.

> Since they involve an exception, they impose an unnecessary "try" on
> all text applications that fear death on bytes returns.  May as well
> just wrap all objects with __fspath__ in fsdecode, and all is
> happy.
>
> Counterproposal: make fsdecode and fsencode grok __fspath__.  Then:

Not being a native English speaker, I'm relying on a Wikipedia
explanation of "grok", but if you mean that fsdecode and fsencode
would accept objects that implement __fspath__, then I think we all
agree on this. Making the stdlib accept path objects, after all, is
the whole point of the pathlib discussions :).

Anyway, I am happy that Nick [3] (and you [4] ?) pointed out that
os.fsencode and os.fsdecode currently implement coercion, i.e., they
both accept both str and bytes, and return just one of them. This was
important for my conclusion in [1]. When these two functions are made
__fspath__ compatible using `fspath(patharg, output_types = (str,
bytes))`, like most os functions, they will indeed implement coercion
to bytes or str from "any pathlike object".

[Side note: One may, for instance, ask why os.fsdecode passes str
objects through silently, even if they can't be decoded. Well, that's
the way it is, and I'm not expecting that to change. But maybe
fsdecode should have an additional keyword-only argument to tell them
that it should strictly return something it actually did decode. (And
similarly for os.fsencode.) But this has nothing to do with the path
protocol we are discussing.]

> (1) Bytes-lovers and str-addicts are both safe.

I don't think everyone is safe if you cant say "I don't want implicit
encoding/decoding".

> (2) They can omit fspath, too!

I think having *one* additonal function for the
non-encoding/non-decoding cases is too much, and as shown in [1], one
is enough.

> No, that doesn't work if the bytes objects aren't in the file system
> encoding, but these are *bytes*, mon ami: you have no way to find out
> what that encoding is, so you either know already and you substitute
> that + fspath for fsdecode, or you're hosed.  And in the only concrete
> use case so far, fsdecode Just Works.

Well, as you say yourself, fsdecode indeed works if your bytes are in
the default fs encoding, and when you know they are, go for it, use
fsdecode. But I, for instance, rarely have my paths as bytes.
Therefore, I would be happy to get an exception if I'm accidentally
passing bytes to some non-bytes-supporting function because I've
forgotten to decode some input that I got in an encoding other than
the file system encoding.

> I suppose a similar argument holds for applications that want bytes
> and fsencode, but I leave that as an exercise for the reader.

A similar counterargument holds, too :).

Unrelated to this particular post, I believe these discussions are
almost done and I truly hope we at least won't have to keep addressing
the same questions that we have already gone through, unless there is
something new on the table.

I hope it takes a shorter time to read these emails than it takes to
write them :).

-Koos

[1] https://mail.python.org/pipermail/python-dev/2016-April/144239.html
[2] https://mail.python.org/pipermail/python-dev/2016-April/144290.html

And somewhat older ones:

[3] https://mail.python.org/pipermail/python-dev/2016-April/144101.html
[4] https://mail.python.org/pipermail/python-dev/2016-April/144107.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion on fspath: please wait for a PEP

[Koos Zevenhoven]

On Wed, Apr 20, 2016 at 7:34 PM, Victor Stinner
 wrote:
>
> 2016-04-20 18:12 GMT+02:00 Brett Cannon :
>>
>> I thought Chris and I w/ Ethan helping with coding, but if it's just me for
>> the PEP then that's fine;

Well, just in case you didn't notice this on python-ideas, I offered
to work on the PEP in case there turns out to be one. This was when
Guido had asked if there is going to be a PEP, in response to my "Type
hinting for path-related functions" email. That offer is certainly
still valid.

>> luckily my firefighter gear is well-worn:
>> https://goo.gl/photos/R8oWdLE45d99ebaw8
>
> LOL, it seems appropriate for this topic...

It sure has been flammable XD

>> I'll try to get a PEP draft written and posted prior to PyCon US. I will
>> reply to any dangling comments/issues that have appeared overnight to close
>> those threads, but otherwise I will start ignoring all discussions so I can
>> focus on the PEP. Everyone can now consider themselves spared from any
>> further path-related discussions. :)

Yes, going from endless discussion to PEP seems like a very healthy
direction at this point.

> I hesitated to propose to create a fspath-sig mailing list, but I suck
> at humor and so I skipped this joke in my email ;-)

You did not have to tell that joke, the joke was present all the time ;).

> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] List posting custom [was: current status of discussions]

[Koos Zevenhoven]

On Wed, Apr 13, 2016 at 5:56 AM, Stephen J. Turnbull  wrote:
> The following is my opinion, as will become obvious, but it's based on
> over a decade of observing these lists, and other open source
> development lists.  In a context where some core developers have
> unsubscribed from these lists, and others regularly report muting
> threads with a certain air of asperity, I think it's worth the risk of
> seeming arrogant to explain some of the customs (which are complex and
> subtle) around posting to Python developer lists.  I'm posting
> publicly because there are several new developers whose activity and
> fresh perspective is very welcome, but harmony *is* being disturbed,
> IMO unnecessarily.
>

Thank you for this thoughtful post. While none of the quotes you refer
to are mine, I did try to find whether any of the advice is something
I should learn from. While I didn't find a whole lot (please do
correct me if you think otherwise), it is also valuable to hear these
things from someone more experienced, even just to confirm what I may
have thought or guessed. I can't really tell, but possibly some of the
thoughts are interesting even to people significantly more experienced
than me.

I know you are not interested in discussing this further here, but
I'll add some inexperienced points of view inline below, just in case
someone is interested:

> This particular post caught my eye, but it's only an example of one of
> the most unharmonious posting styles that has become common recently.
> Attribution deliberately removed.
>
>  > Sorry for disturbing this thread's harmony.
>
> *sigh*  There is way too much of this on Python-Ideas recently, and
> there shouldn't be any on Python-Dev.  So please don't.  Specifically,
> disagreement with an apparently developing consensus is fine but
> please avoid this:
>
>  > >> Path is an alternative to os.path -- you don't need to use both.
>  >
>  > I agree with that quote of Chris.
>
> It's a waste of time to post *what* you agree with.[1]  Decisions are
> not taken by vote in this community, except for the color of the
> bikeshed, where it is agreed that *what* decision is taken doesn't
> matter, but that some decision should be taken expeditiously.[2]
> Chris already stated this position clearly and it's not a "color", so
> there is no need to reiterate.  It simply wastes others' time to read
> it.  (Whether it was a waste of the poster's time is not for me to
> comment on.)
>
> What matters to the decision is *why* you agree (or disagree).  If you
> think that some of Chris's arguments are bogus (and should be
> disregarded) and others are important, that is valuable information.
> It's even better if you can shed additional light on the matter
> (example below).
>
> Also, expression of agreement is often a prelude to a request for
> information.  "I agree with Z's post.  At least, I have never needed
> X.  *When* do you need X?  Let's look for a better way than X!"
>

That's what I thought too. I remember several times recently that I
have mentioned I agreed about something, then continuing to add more
to it, or even saying I disagree about something else. Part of the
reason to also state that I agree is an attempt to keep the overall
tone more positive. After all, the other person might be a highly
experienced core developer who just did not happen to have gone though
all the same thoughts regarding that specific question recently. I
hope that has not been interpreted as arrogance such as "I know better
than these people".

For me, as one of the (many?) newcomers, especially on -dev, it can
sometimes be difficult to tell whether not getting a reaction means
"Good point, I agree", "I did not understand so I'll just ignore it",
"I don't want to argue with you" or something else. Then again,
someone just saying essentially the same thing without a reference a
few posts later just feels strange. Also, if the only thing people
apparently do is disagree about things, it makes the overall tone of
the discussions at least *seem* very negative. From this point of view
there seems to be some good in positive comments.

> Unsupported (dis)agreement to statements about "needs" also may be
> taken as *rude*, because others may infer your arrogant claim to know
> what *they* do or don't need.  Admittedly there's a difficult
> distinction here between Chris's *idiom* where "you don't need to"
> translates to "In my understanding, it is generally not necessary to",
> and your *unsupported* agreement, which in my dialect of English
> changes the emphasis to imply you know better than those who disagree
> with you and Chris.  And, of course, the position that others are "too
> easily offended" is often reasonable, but you should be aware that
> there will be an impact on your reputation and ability to influence
> development of Python (even if it doesn't come near the point where
> a moderator invokes "Code of Conduct").
>
> "Me too" posts aren't entirely 

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Koos Zevenhoven]

On Mon, Apr 18, 2016 at 5:03 PM, Ethan Furman  wrote:
> On 04/18/2016 12:41 AM, Nick Coghlan wrote:
>
>> Given the variant you [Koos] suggested, what if we defined the API
>> semantics
>> like this:
>>
>>  # Offer the simplest possible API as the public vesion
>>  def fspath(pathlike) -> str:
>>  return os._raw_fspath(pathlike)
>>
>>  # Expose the complexity in the "private" variant
>>  def _raw_fspath(pathlike, *, output_types = (str,)) -> (str, bytes):
>>  # Short-circuit for instances of the output type
>>  if isinstance(pathlike, output_types):
>>  return pathlike
>>  # We'd have a tidier error message here for non-path objects
>>  result = pathlike.__fspath__()
>>  if not isinstance(result, output_types):
>>  raise TypeError("argument is not and does not provide an
>> acceptable pathname")
>>  return result
>
> My initial reaction was that this was overly complex, but after thinking
> about it a couple days I /really/ like it.  It has a reasonable default for
> the 99% real-world use-case, while still allowing for custom and exact
> tailoring (for the 99% stdlib use-case ;) .
>

While it does seem we finally might be nearly there :), this still
seems to need some further discussion.

As described in that long post of mine, I suppose some third-party
code may need the variations (A-C), while it seems that in the stdlib,
most places need (str, bytes), i.e. (A), except in pathlib, which
needs (str,), i.e. (B). I'm not sure what I think about making the
variations private, even if "hiding" the bytes version is, as I said,
an important role of the public function.

Except for that type hint, there is *nothing* in the function that
might mislead the user to think bytes paths are something important in
Python 3. It's a matter of documentation whether it "supports" bytes
or not. In fact, that function (assuming the name os.fspath) could now
even be documented to support this:

patharg = os.fspath(patharg, output_types = (str, pathlib.PurePath))  # :-)

So are we still going to end up with two functions or can we deal with one?
What should the typehint be? Something new in typing.py? How about
FSPath[...] as follows:

FSPath[bytes]  # bytes-based pathlike, including bytes
FSPath[str]   # str-based pathlike, including str

pathstring = typing.TypeVar('pathstring', str, bytes)  # could be
extended with PurePath or some path ABC

So the above variation might become:

def fspathname(pathlike: FSPath[pathstring],
   *, output_types: tuple = (str,)) -> pathstring:
# Short-circuit for instances of the output type
if isinstance(pathlike, output_types):
return pathlike
# We'd have a tidier error message here for non-path objects
result = pathlike.__fspath__()
if not isinstance(result, output_types):
raise TypeError("valid output type not provided via __fspath__")
return result

And similar type hints would apply to os.path functions. For instance,
os.path.dirname:

def dirname(p: FSPath[pathstring]) -> pathstring:
...

This would say pathstring all over and not give anyone any ideas about
bytes, unless they know what they're doing.

Complicated? Yes, typing is. But I think we will need this kind of
hints for os.path functions anyway.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Koos Zevenhoven]

On Thu, Apr 14, 2016 at 7:46 PM, Ethan Furman  wrote:
>
> What many folks seem to be missing is that *you* (generic you) have control
> of your data.
>
> If you are not working at the bytes layer, you shouldn't be getting bytes
> objects because:
>
> - you specified str when asking for data from the OS, or
> - you transformed the incoming bytes from whatever external source
>   to str when you received them.

There is an apparent contradiction of the above with some previous
posts, including your own. Let me try to fix it:

Code that deals with paths can be divided in groups as follows:

(1) Code that has access to pathname/filename data and has some level
of control over what data type comes in. This code may for instance
choose to deal with either bytes or str

(2) Code that takes the path or file name that it happens to get and
does something with it. This type of code can be divided into
subgroups as follows:

  (2a) Code that accepts only one type of paths (e.g. str, bytes or
pathlib) and fails if it gets something else.

  (2b) Code that wants to support different types of paths such as
str, bytes or pathlib objects. This includes os.path.*, os.scandir,
and various other standard library code. Presumably there is also
third-party code that does the same. These functions may want to
preserve the str-ness or bytes-ness of the paths in case they return
paths, as the stdlib now does. But new code may even want to return
pathlib objects when they get such objects as inputs. This is the
duck-typing or polymorphic code we have been talking about. Code of
this type (2b) may want to avoid implicit conversions because it makes
the life of code of the other types more difficult.

(feel free to fill in more categories of code)

So the code of type (2b) is trying to make all categories happy by
returning objects of the same type that it gets as input, while the
other categories are probably in the situation where they don't
necessarily need to make other categories of code happy.

And the question is this: Do we need to make code using both bytes
*and* scandir happy? This is largely the same question as whether we
have to support bytes in addition to str in the protocol.

(We may of course talk about third-party path libraries that have the
same problem as scandir's DirEntry. Ethan's library is not exactly in
the same category as DirEntry since its path objects *are* instances
of bytes or str and therefore do not need this protocol to begin with,
except perhaps for conversions from other high-level path types so
that different path libraries work together nicely).

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

tr = 'Picard'
> damage: int
> stats: ClassVar[Dict[str, int]] = {}
>
> def __init__(self, damage: int, captain: str = None):
> self.damage = damage
> if captain:
> self.captain = captain  # Else keep the default
>
> With type hints as they currently exist, the same code is shorter and
> doesn't contaminate the class namespace with the 'damage' attribute.

IIUC, 'damage' will not be in the class namespace according to PEP 526.

> class Starship:
>
> captain = 'Picard'
> stats = {} # type: Dict[str, int]
>
> def __init__(self, damage: int, captain: str = None):
> self.damage = damage # Can infer type as int
> if captain:
> self.captain = captain # Can infer type as str
>

And that's one of the reasons why there should be annotations without
setting a type hint (as I wrote in the other thread).

>
> This isn't an argument against adding type syntax for attributes in general,
> just that the form suggested in PEP 526 doesn't seem to follow Python
> semantics.
>
> One could imagine applying minimal PEP 526 style hints, with standard Python
> semantics and relying on type inference, as follows:
>
> class Starship:
>
> captain = 'Picard'
> stats: Dict[str, int] = {}
>
> def __init__(self, damage: int, captain: str = None):
> self.damage = damage
> if captain:
> self.captain = captain

I don't like this, because some of the attributes are introduced at
class level and some inside __init__, so it is easy to miss that there
is such a thing as 'damage' (at least in more complicated examples). I
keep repeating myself, but again this where we need non-type-hinting
attribute declarations.

-- Koos

>
> The PEP overstates the existing use of static typing in Python
> ==
>
[...]
> Please don't turn Python into some sort of inferior Java.
> There is potential in this PEP, but in its current form I think it should be
> rejected.
>
> Cheers,
> Mark.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

t' object cannot be interpreted as an integer
>
> While this does create some duplication, there are valid reasons for it::
>
> * the ``bchr`` builtin is to recreate the ord/chr/unichr trio from Python
> 2 under a different naming scheme
> * the class method is mainly for the ``bytearray.fromord`` case, with
> ``bytes.fromord`` added for consistency
>
> The documentation of the ``ord`` builtin will be updated to explicitly
note
> that ``bchr`` is the primary inverse operation for binary data, while
> ``chr``
> is the inverse operation for text data, and that ``bytes.fromord`` and
> ``bytearray.fromord`` also exist.
>
> Behaviourally, ``bytes.fromord(x)`` will be equivalent to the current
> ``bytes([x])`` (and similarly for ``bytearray``). The new spelling is
> expected to be easier to discover and easier to read (especially when used
> in conjunction with indexing operations on binary sequence types).
>
> As a separate method, the new spelling will also work better with higher
> order functions like ``map``.
>
>
> Addition of "getbyte" method to retrieve a single byte
> --
>
> This PEP proposes that ``bytes`` and ``bytearray`` gain the method
> ``getbyte``
> which will always return ``bytes``::
>
> >>> b'abc'.getbyte(0)
> b'a'
>
> If an index is asked for that doesn't exist, ``IndexError`` is raised::
>
> >>> b'abc'.getbyte(9)
> Traceback (most recent call last):
> File "", line 1, in 
> IndexError: index out of range
>
>
> Addition of optimised iterator methods that produce ``bytes`` objects
> -
>
> This PEP proposes that ``bytes`` and ``bytearray``gain an optimised
> ``iterbytes`` method that produces length 1 ``bytes`` objects rather than
> integers::
>
> for x in data.iterbytes():
> # x is a length 1 ``bytes`` object, rather than an integer
>
> For example::
>
> >>> tuple(b"ABC".iterbytes())
> (b'A', b'B', b'C')
>
>
> Design discussion
> =
>
> Why not rely on sequence repetition to create zero-initialised sequences?
> -
>
> Zero-initialised sequences can be created via sequence repetition::
>
> >>> b'\x00' * 3
> b'\x00\x00\x00'
> >>> bytearray(b'\x00') * 3
> bytearray(b'\x00\x00\x00')
>
> However, this was also the case when the ``bytearray`` type was originally
> designed, and the decision was made to add explicit support for it in the
> type constructor. The immutable ``bytes`` type then inherited that feature
> when it was introduced in PEP 3137.
>
> This PEP isn't revisiting that original design decision, just changing the
> spelling as users sometimes find the current behaviour of the binary
> sequence
> constructors surprising. In particular, there's a reasonable case to be
made
> that ``bytes(x)`` (where ``x`` is an integer) should behave like the
> ``bytes.fromint(x)`` proposal in this PEP. Providing both behaviours as
> separate
> class methods avoids that ambiguity.
>
>
> Open Questions
> ==
>
> Do we add ``iterbytes`` to ``memoryview``, or modify
> ``memoryview.cast()`` to accept ``'s'`` as a single-byte interpretation?
Or
> do we ignore memory for now and add it later?
>
>
> References
> ==
>
> .. [1] Initial March 2014 discussion thread on python-ideas
> (https://mail.python.org/pipermail/python-ideas/2014-March/027295.html)
> .. [2] Guido's initial feedback in that thread
> (https://mail.python.org/pipermail/python-ideas/2014-March/027376.html)
> .. [3] Issue proposing moving zero-initialised sequences to a dedicated
API
> (http://bugs.python.org/issue20895)
> .. [4] Issue proposing to use calloc() for zero-initialised binary
sequences
> (http://bugs.python.org/issue21644)
> .. [5] August 2014 discussion thread on python-dev
> (https://mail.python.org/pipermail/python-ideas/2014-March/027295.html)
> .. [6] June 2016 discussion thread on python-dev
> (https://mail.python.org/pipermail/python-dev/2016-June/144875.html)
>
>
> Copyright
> =
>
> This document has been placed in the public domain.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526)

[Koos Zevenhoven]

On Fri, Sep 2, 2016 at 9:04 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> On Fri, Sep 02, 2016 at 08:10:24PM +0300, Koos Zevenhoven wrote:
>
>> A good checker should be able to infer that x is a union type at the
>> point that it's passed to spam, even without the type annotation. For
>> example:
>>
>> def eggs(cond:bool):
>> if cond:
>> x = 1
>> else:
>> x = 1.5
>> spam(x)   # a good type checker infers that x is of type Union[int, 
>> float]
>
> Oh I really hope not. I wouldn't call that a *good* type checker. I
> would call that a type checker that is overly permissive.

I guess it's perfectly fine if we disagree about type checking ideals,
and I can imagine the justification for you thinking that way. There
can also be different type checkers, and which can have different
modes.

But assume (a) that the above function is perfectly working code, and
spam(...) accepts Union[int, float]. Why would I want the type checker
to complain?

Then again, (b) instead of that being working code, it might be an
error and spam only takes float. No problem, the type checker will
catch that.

In case of (b), to get the behavior you want (but in my hypothetical
semantics), this could be annotated as

def eggs(cond:bool):
x : float
if cond:
x = 1  # type checker says error
else:
x = 1.5
spam(x)

So here the programmer thinks the type of x should be more constrained
than what spam(...) accepts.

Or you might have something like this

def eggs(cond:bool):
if cond:
x = 1
else:
x = 1.5
# type checker has inferred x to be Union[int, float]
x : float  # type checker finds an error
spam(x)

Here, the same error is found, but at a different location.

> Maybe you think that it's okay because ints and floats are somewhat
> compatible. But suppose I wrote:
>
> if cond:
> x = HTTPServer(*args)
> else:
> x = 1.5

It might be clear by now, but no, that's not why I wrote that. That
was just a slightly more "realistic" example than this HTTP & 1.5 one.

[...]
> Do you have a better idea for variable
> syntax?

I had one but it turned out it was worse.

-- Koos

>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 1:04 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 5 September 2016 at 18:19, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>> On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
>>> On 5 September 2016 at 04:40, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>>>> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi <levkivs...@gmail.com> 
>>>> wrote:
>>>>> On 4 September 2016 at 19:59, Nick Coghlan <ncogh...@gmail.com> wrote:
>>>> [...]
>>>>>>
>>>>>> Similarly, it would be reasonable to say that these three snippets
>>>>>> should all be equivalent from a typechecking perspective:
>>>>>>
>>>>>> x = None # type: Optional[T]
>>>>>>
>>>>>> x: Optional[T] = None
>>>>>>
>>>>>> x: Optional[T]
>>>>>> x = None
>>>>>
>>>>>
>>>>> Nice idea, explicit is better than implicit.
>>>>
>>>> How is it going to help that these are equivalent within one checker,
>>>> if the meaning may differ across checkers?
>>>
>>> For typechecker developers, it provides absolute clarity that the
>>> semantics of the new annotations should match the behaviour of
>>> existing type comments when there's an initialiser present,
>>
>> I understood that, but what's the benefit? I hope there will be a type
>> checker that breaks this "rule".
>
> Such a typechecker means you're not writing Python anymore, you're
> writing Java/C++/C# in a language that isn't designed to be used that
> way.

I'm glad those are all the languages you accuse me of. The list could
have been a lot worse. I actually have some good memories of Java. It
felt kind of cool at that age, and it taught me many things about
undertanding the structure of large and complicated programs after I
had been programming for years in other languages, including C++. It
also taught me to value simplicity instead, so here we are.

> Fortunately, none of the current typecheckers have made that mistake,
> nor does anyone appear to be promoting this mindset outside this
> particular discussion.

The thing I'm promoting here is to not add anything to PEP 526 that
says what a type checker is supposed to do with type annotations.
Quite the opposite of Java/C++/C#, I would say.

We can, of course, speculate about the future of type checkers and the
implications of PEP 526 on it. That's what I'm trying to do on
python-ideas, speculate about the best kind of type checking
(achievable with PEP 526 annotations) [1].

--Koos


[1] https://mail.python.org/pipermail/python-ideas/2016-September/042076.html

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526)

[Koos Zevenhoven]

What's up with the weird subthreads, Stephen?!

On Guido's suggestion, I'm working on posting those type-checking thoughts here.

-- Koos

On Sat, Sep 3, 2016 at 6:17 PM, Stephen J. Turnbull
<turnbull.stephen...@u.tsukuba.ac.jp> wrote:
> Please respect Reply-To, set to python-ideas.
>
> Greg Ewing writes:
>  > Chris Angelico wrote:
>  > > Forcing people to write 1.0 just to be compatible with 1.5 will cause
>  > > a lot of annoyance.
>  >
>  > Indeed, this would be unacceptable IMO.
>
> But "forcing" won't happen.  Just ignore the warning.  *All* such
> Python programs will continue to run (or crash) exactly as if the type
> declarations weren't there.  If you don't like the warning, either
> don't run the typechecker, or change your code to placate it.
>
> But allowing escapes from a typechecker means allowing escapes.  All
> of them, not just the ones you or I have preapproved.  I want my
> typechecker to be paranoid, and loud about it.
>
> That doesn't mean I would never use a type like "Floatable" (ie, any
> type subject to implicit conversion to float).  But in the original
> example, I would probably placate the typechecker.  YMMV, of course.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 1:23 AM, Ivan Levkivskyi <levkivs...@gmail.com> wrote:
> On 4 September 2016 at 00:11, Random832 <random...@fastmail.com> wrote:
>>
>> On Sat, Sep 3, 2016, at 18:06, Koos Zevenhoven wrote:
>> > I guess one reason I don't like bchr (nor chrb, really) is that they
>> > look just like a random sequence of letters in builtins, but not
>> > recognizable the way asdf would be.
>> >
>> > I guess I have one last pair of suggestions for the name of this
>> > function: bytes.chr or bytes.char.
>>
>> What about byte? Like, not bytes.byte, just builtins.byte.
>
>
> I like this option, it would be very "symmetric" to have, compare:
>
>>>>chr(42)
> '*'
>>>>str()
> ''
>
> with this:
>
>>>>byte(42)
> b'*'
>>>>bytes()
> b''
>
> It is easy to explain and remember this.

In one way, I like it, but on the other hand, indexing a bytes gives
an integer, so maybe a 'byte' is just an integer in range(256). Also,
having both byte and bytes would be a slight annoyance with
autocomplete.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

On Sat, Sep 3, 2016 at 7:59 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 3 September 2016 at 03:54, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>> chrb seems to be more in line with some bytes versions in for instance os
>> than bchr.
>
> The mnemonic for the current name in the PEP is that bchr is to chr as
> b"" is to "". The PEP should probably say that in addition to pointing
> out the 'unichr' Python 2 inspiration, though.

Thanks for explaining. Indeed I hope that unichr does not affect any
naming decisions that will remain in the language for a long time.

> The other big difference between this and the os module case, is that
> the resulting builtin constructor pairs here are str/chr (arbitrary
> text, single code point) and bytes/bchr (arbitrary binary data, single
> binary octet). By contrast, os.getcwd() and os.getcwdb() (and similar
> APIs) are both referring to the same operating system level operation,
> they're just requesting a different return type for the data.

But chr and "bchr" are also requesting a different return type. The
difference is that the data is not coming from an os-level operation
but from an int.

I guess one reason I don't like bchr (nor chrb, really) is that they
look just like a random sequence of letters in builtins, but not
recognizable the way asdf would be.

I guess I have one last pair of suggestions for the name of this
function: bytes.chr or bytes.char.

-- Koos


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

On Sat, Sep 3, 2016 at 6:41 PM, Ethan Furman <et...@stoneleaf.us> wrote:
>>>
>>> Open Questions
>>> ==
>>>
>>> Do we add ``iterbytes`` to ``memoryview``, or modify
>>> ``memoryview.cast()`` to accept ``'s'`` as a single-byte interpretation?
>>> Or
>>> do we ignore memory for now and add it later?
>>
>>
>> Apparently memoryview.cast('s') comes from Nick Coghlan:
>>
>> <https://marc.info/?i=CADiSq7e=8ieyew-txf5dims_5nuaos5udv-3g_w3ltwn9wb...@mail.gmail.com>.
>> However, since 3.5 (https://bugs.python.org/issue15944) you can call
>> cast("c") on most memoryviews, which I think already does what you
>> want:
>>
>>>>> tuple(memoryview(b"ABC").cast("c"))
>>
>> (b'A', b'B', b'C')
>
>
> Nice!
>

Indeed! Exposing this as bytes_instance.chars would make porting from
Python 2 really simple. Of course even better would be if slicing the
view would return bytes, so the porting rule would be the same for all
bytes subscripting:

py2str[SOMETHING]

becomes

py3bytes.chars[SOMETHING]

With the "c" memoryview there will be a distinction between slicing
and indexing.

And Random832 seems to be making some good points.

--- Koos


> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 5:02 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 5 September 2016 at 23:46, Nick Coghlan <ncogh...@gmail.com> wrote:
>> Under such "parameter annotation like" semantics, uninitialised
>> variable annotations would only make sense as a new form of
>> post-initialisation assertion,

Why not discuss this in the python-ideas thread where I quote myself
from last Friday regarding the notion of annotations as assertions?

>> and perhaps as some form of
>> Eiffel-style class invariant documentation syntax.

I hope this is simpler than it sounds :-)

> Thinking further about the latter half of that comment, I realised
> that the PEP 484 equivalence I'd like to see for variable annotations
> in a class body is how they would relate to a property definition
> using the existing PEP 484 syntax.
>
> For example, consider:
>
> class AnnotatedProperty:
>
> @property
> def x(self) -> int:
> ...
>
> @x.setter
> def x(self, value: int) -> None:
> ...
>
> @x.deleter
> def x(self) -> None:
> ...
>
> It would be rather surprising if that typechecked differently from:
>
> class AnnotatedVariable:
>
> x: int
>

How about just using the latter way? That's much clearer. I doubt this
needs a change in the PEP.

> For ClassVar, you'd similarly want:
>
>
> class AnnotatedClassVariable:
>
> x: ClassVar[int]
>
> to typecheck like "x" was declared as an annotated property on the metaclass.
>

Sure, there are many things that one may consider equivalent. I doubt
you'll be able to list them all in a way that everyone agrees on. And
I hope you don't take this as a challenge -- I'm in the don't-panic
camp :).


-- Koos


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

It looks like you are trying to make sense of this, but unfortunately
there's some added mess and copy errors regarding who said
what. I think no such errors remain in what I quote below:

On Mon, Sep 5, 2016 at 3:10 PM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> [Koos Zevenhoven]
>> >> How is it going to help that these are equivalent within one checker,
>> >> if the meaning may differ across checkers?
>
> Before I can give an answer to your [Koos'] question, I have to
> understand what you see as the problem here.

The problem was that suggested restrictive addition into PEP 526 with
no proper justification, especially since the PEP was not supposed to
restrict the semantics of type checking. I was asking how it would
help to add that restriction. Very simple. Maybe some people got
confused because I did want to *discuss* best practices for type
checking elsewhere.

> I *think* that you are worried that two different checkers will disagree
> on what counts as a type error. That given the same chunk of code:

In the long term, I'm worried about that, but there's nothing that PEP
526 can do about it at this point.

> [Nick Coghlan]
>> > For typechecker developers, it provides absolute clarity that the
>> > semantics of the new annotations should match the behaviour of
>> > existing type comments when there's an initialiser present,
>
> [Koos]
>> I understood that, but what's the benefit?
>
> Are you asking what is the benefit of having three forms of syntax for
> the same thing?

No, still the same thing: What is the benefit of that particular
restriction, when there are no other restrictions? Better just leave
it out.

> The type comment systax is required for Python 2 and backwards-
> compatibility. That's a given.

Sure, but all type checkers will not have to care about Python 2.

> The variable annotation syntax is required because the type comment
> syntax is (according to the PEP) very much a second-best solution. See
> the PEP:
>
> https://www.python.org/dev/peps/pep-0526/#id4
>
> So this is a proposal to create a *better* syntax for something which
> already exists. The old version, using comments, cannot be deprecated or
> removed, as it is required for Python 3.5 and older.

Right.

> Once we allow
>
> x: T = value
>
> then there benefit in also allowing:
>
> x: T
> x = value
>
> since this supports some of the use cases that aren't well supported by
> type comments or one-line variable annotations. E.g. very long or deeply
> indented lines, situations where the assignment to x is inside an
> if...else branch, or any other time you wish to declare the type of the
> variable before actually setting the variable.

Sure.

> [Nick]
>> > For folks following along without necessarily keeping up with all the
>> > nuances, it makes it more explicit what Guido means when he says "PEP
>> > 526 does not make a stand on the
>> > behavior of type checkers (other than deferring to PEP 484)."
>
> [Koos]
>> What you are proposing is exactly "making a stand on the behavior of
>> type checkers", and the examples you provide below are all variations
>> of the same situation and provide no justification for a general rule.
>
> I'm sorry, I don't understand this objection. The closest I can get to
> an answer would be:
>
> A general rule is better than a large number of unconnected, arbitrary,
> special cases.

A general rule that does not solve a problem is worse than no rule.


-- Koos


>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

Sorry, I don't have time to read emails of this length now, and
perhaps I'm interpreting your emails more literally than you write
them, anyway.

If PEP 484 introduces unnecessary restrictions at this point, that's a
separate issue. I see no need to copy those into PEP 526. I'll be
posting my own remaining concerns regarding PEP 526 when I find the
time.

-- Koos


On Mon, Sep 5, 2016 at 4:46 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 5 September 2016 at 21:46, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>> The thing I'm promoting here is to not add anything to PEP 526 that
>> says what a type checker is supposed to do with type annotations.
>
> PEP 526 says it doesn't intend to expand the scope of typechecking
> semantics beyond what PEP 484 already supports. For that to be true,
> it needs to be able to define expected equivalencies between the
> existing semantics of PEP 484 and the new syntax in PEP 526.
>
> If those equivalencies can't be defined, then Mark's concerns are
> valid, and the PEP either needs to be deferred as inadvertently
> introducing new semantics while intending to only introduce new
> syntax, or else the intended semantics need to be spelled out as they
> were in PEP 484 so folks can judge the proposal accurately, rather
> than attempting to judge it based on an invalid premise.
>
> For initialised variables, the equivalence between the two PEPs is
> straightforward: "x: T = expr" is equivalent to "x = expr # type: T"
>
> If PEP 526 always required an initialiser, and didn't introduce
> ClassVar, there'd be no controversy, and we'd already be done.
>
> However, the question of "Does this new syntax necessarily imply the
> introduction of new semantics?" gets a lot murkier for uninitialised
> variables.
>
> A strict "no new semantics beyond PEP 484" interpretation would mean
> that these need to be interpreted the same way as parameter
> annotations: as a type hint on the outcome of the code executed up to
> that point, rather than as a type constraint on assignment statements
> in the code *following* that point.
>
> Consider:
>
> def simple_appender(base: List[T], value: T) -> None:
> base.append(value)
>
> This will typecheck fine - lists have append methods, and the value
> appended conforms to what our list expects.
>
> The parameter annotations mainly act as constraints on how this
> function is *called*, with the following all being problematic:
>
> simple_appender([1, 2, 3], "hello") # Container/value type mismatch
> simple_appender([1, 2, 3], None) # Value is not optional
> simple_appender((1, 2, 3), 4) # A tuple is not a list
>
> However, because of the way name binding in Python works, the
> annotations in *no way* constrain assignments inside the function
> body:
>
> def not_so_simple_appender(base: List[T], value: T) -> None:
> other_ref = base
> base = value
> other_ref.append(base)
>
> From a dynamic typechecking perspective, that's just as valid as the
> original implementation, since the "List[T]" type of "other_ref" is
> inferred from the original type of "base" before it gets rebound to
> value and has its new type inferred as "T".
>
> This freedom to rebind an annotated name without a typechecker
> complaining is what Mark is referring to when he says that PEP 484
> attaches annotations to expressions rather than types.
>
> Under such "parameter annotation like" semantics, uninitialised
> variable annotations would only make sense as a new form of
> post-initialisation assertion, and perhaps as some form of
> Eiffel-style class invariant documentation syntax.
>
> The usage to help ensure code correctness in multi-branch
> initialisation cases would then look something like this:
>
>if case1:
> x = ...
> elif case2:
> x = ...
> else:
> x = ...
> assert x : List[T] # If we get to here without x being List[T],
> something's wrong
>
> The interpreter could then optimise type assertions out entirely at
> function level (even in __debug__ mode), and turn them into
> annotations at module and class level (with typecheckers then deciding
> how to process them).
>
> That's not what the PEP proposes for uninitialised variables though:
> it proposes processing them *before* a series of assignment
> statements, which *only makes sense* if you plan to use them to
> constrain those assignments in some way.
>
> If you wanted to write something like that under a type assertion
> spelling, then you could enlist the aid of the "all" builtin:
>
> assert all(x) : Li

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Koos Zevenhoven]

On Tue, Sep 6, 2016 at 1:49 AM, Sven R. Kunze <srku...@mail.de> wrote:
> Didn't Koos say this works more like an expression annotation?
>
> IMO, the type of the expression is what is specified but the type of the
> variable can change over time (as you demonstrated).

That's exactly the kind of semantics I'm describing in the
python-ideas thread. An that's exactly how Python works: the type of a
variable can change every time you assign a value to it (but not in
between, unless you're doing funny stuff). So in a sense you annotate
the *value* by annotating the variable at the point in the function
where the value is assigned to it.

There are open questions in this approach of course. But if you're
interested, don't hesitate to discuss or ask questions in the
python-ideas thread. I won't answer before I wake up, though ;).

-- Koos


>
> Sven
>
>
> PS: thinking this way, the new syntax is actually confusing as it annotates
> the variable not the expression. :-/
>
>
>
> On 05.09.2016 17:26, Mark Shannon wrote:
>>
>> Hi,
>>
>> PEP 526 states that "This PEP aims at adding syntax to Python for
>> annotating the types of variables" and Guido seems quite insistent that the
>> declarations are for the types of variables.
>>
>> However, I get the impression that most (all) of the authors and
>> proponents of PEP 526 are quite keen to emphasise that the PEP in no way
>> limits type checkers from doing what they want.
>>
>> This is rather contradictory. The behaviour of a typechecker is defined by
>> the typesystem that it implements. Whether a type annotation determines the
>> type of a variable or an expression alters changes what typesystems are
>> feasible. So, stating that annotations define the type of variables *does*
>> limit what a typechecker can or cannot do.
>>
>> Unless of course, others may have a different idea of what the "type of a
>> variable" means.
>> To me, it means it means that for all assignments `var = expr`
>> the type of `expr` must be a subtype of the variable,
>> and for all uses of var, the type of the use is the same as the type of
>> the variable.
>>
>> In this example:
>>
>> def bar()->Optional[int]: ...
>>
>> def foo()->int:
>> x:Optional[int] = bar()
>> if x is None:
>> return -1
>> return x
>>
>> According to PEP 526 the annotation `x:Optional[int]`
>> means that the *variable* `x` has the type `Optional[int]`.
>> So what is the type of `x` in `return x`?
>> If it is `Optional[int]`, then a type checker is obliged to reject this
>> code. If it is `int` then what does "type of a variable" actually mean,
>> and why aren't the other uses of `x` int as well?
>>
>> Cheers,
>> Mark.
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>
>
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 5 September 2016 at 04:40, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi <levkivs...@gmail.com> wrote:
>>> On 4 September 2016 at 19:59, Nick Coghlan <ncogh...@gmail.com> wrote:
>> [...]
>>>>
>>>> Similarly, it would be reasonable to say that these three snippets
>>>> should all be equivalent from a typechecking perspective:
>>>>
>>>> x = None # type: Optional[T]
>>>>
>>>> x: Optional[T] = None
>>>>
>>>> x: Optional[T]
>>>> x = None
>>>
>>>
>>> Nice idea, explicit is better than implicit.
>>
>> How is it going to help that these are equivalent within one checker,
>> if the meaning may differ across checkers?
>
> For typechecker developers, it provides absolute clarity that the
> semantics of the new annotations should match the behaviour of
> existing type comments when there's an initialiser present,

I understood that, but what's the benefit? I hope there will be a type
checker that breaks this "rule".

> or of a
> parameter annotation when there's no initialiser present.

No, your suggested addition does not provide any reference to this.
(...luckily, because that would have been worse.)

> For folks following along without necessarily keeping up with all the
> nuances, it makes it more explicit what Guido means when he says "PEP
> 526 does not make a stand on the
> behavior of type checkers (other than deferring to PEP 484)."

What you are proposing is exactly "making a stand on the behavior of
type checkers", and the examples you provide below are all variations
of the same situation and provide no justification for a general rule.

Here's a general rule:

The closer it gets to the end of drafting a PEP [1],
the more carefully you have to justify changes.

Justification is left as an exercise ;-).

--Koos

[1] or any document (or anything, I suppose)

> For example, the formulation of the divergent initialisation case
> where I think the preferred semantics are already implied by PEP 484
> can be looked at this way:
>
> x = None # type: Optional[List[T]]
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> It would be a strange typechecker indeed that handled that case
> differently from the new spellings made possible by PEP 526:
>
> x: Optional[List[T]] = None
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> x: Optional[List[T]]
> if arg is None:
> x = None
> else:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> x: Optional[List[T]]
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
> else:
> x = None
>
> Or from the semantics of PEP 484 parameter annotations:
>
> def my_func(arg:Optional[List[T]], other_arg=None):
> # other_arg is implicitly Optional[Any]
> if arg is not None and other_arg is not None:
> # Here, "arg" can be assumed to be List[T]
> # while "other_arg" is Any
> arg.extend(other_arg)
>
> A self-consistent typechecker will either allow all of the above, or
> prohibit all of the above, while a typechecker that *isn't*
> self-consistent would be incredibly hard to use.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 3:30 AM, Random832 <random...@fastmail.com> wrote:
> On Sun, Sep 4, 2016, at 16:42, Koos Zevenhoven wrote:
>> On Sun, Sep 4, 2016 at 6:38 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
>> >
>> > There are two self-consistent sets of names:
>> >
>>
>> Let me add a few. I wonder if this is really used so much that
>> bytes.chr is too long to type (and you can do bchr = bytes.chr if you
>> want to):
>>
>> bytes.chr (or bchr in builtins)
>> bytes.chr_at, bytearray.chr_at
>
> Ugh, that "at" is too reminiscent of java. And it just feels wrong to
> spell it "chr" rather than "char" when there's a vowel elsewhere in the
> name.
>

Oh, I didn't realize that connection. It's funny that I get a Java
connotation from get* methods ;).


> Hmm... how offensive to the zen of python would it be to have "magic" to
> allow both bytes.chr(65) and b'ABCDE'.chr[0]? (and possibly also
> iter(b'ABCDE'.chr)? That is, a descriptor which is callable on the
> class, but returns a view on instances?

Indeed quite magical, while I really like how easy it is to remember
this *once you realize what is going on*.

I think bytes.char (on class) and data.chars (on instance) would be
quite similar.

-- Koos


> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 6:06 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 5 September 2016 at 06:42, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>> On Sun, Sep 4, 2016 at 6:38 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
>>>
>>> There are two self-consistent sets of names:
>>>
>>
>> Let me add a few. I wonder if this is really used so much that
>> bytes.chr is too long to type (and you can do bchr = bytes.chr if you
>> want to)
>>
>> bytes.chr (or bchr in builtins)
>
> The main problem with class method based spellings is that we need to
> either duplicate it on bytearray or else break the bytearray/bytes
> symmetry and propose "bytearray(bytes.chr(x))" as the replacement for
> current cryptic "bytearray([x])"

Warning: some API-design philosophy below:

1. It's not as bad to break symmetry regarding what functionality is
offered for related object types (here: str, bytes, bytearray) than it
is to break symmetry in how the symmetric functionality is provided.
IOW, a missing unnecessary functionality is less bad than exposing the
equivalent functionality under a different name. (This might be kind
of how Random832 was reasoning previously)

2. Symmetry is more important in object access functionality than it
is in instance creation. IOW, symmetry regarding 'constructors' (here:
bchr, bytes.chr, bytes.byte, ...) across different types is not as
crucial as symmetry in slicing. The reason is that the caller of a
constructor is likely to know which class it is instantiating. A
consumer of bytes/bytearray/str-like objects often does not know which
type is being dealt with.


I might be crying over spilled milk here, but that seems to be the
point of the whole PEP. That chars view thing might collect some of
the milk back back into a bottle:

mystr[whatever]  <-> mybytes.chars[whatever] <-> mybytearray.chars[whatever]
iter(mystr) <-> iter(mybytes.chars) <-> iter(mybytearray.chars)

Then introduce 'chars' on str and this becomes

mystring.chars[whatever]  <-> mybytes.chars[whatever] <->
mybytearray.chars[whatever]
iter(mystr.chars) <-> iter(mybytes.chars) <-> iter(mybytearray.chars)

If iter(mystr.chars) is recommended and iter(mystr) discouraged, then
after a decade or two, the world may look quite different regarding
how important it is for a str to be iterable.

This would solve multiple problems at once. Well I admit that "at
once" is not really an accurate description of the process :).

[...]
> You also run into a searchability problem as "chr" will get hits for
> both the chr builtin and bytes.chr, similar to the afalg problem that
> recently came up in another thread. While namespaces are a honking
> great idea, the fact that search is non-hierarchical means they still
> don't give API designers complete freedom to reuse names at will.

Oh, I can kind of see a point here, especially if the search hits
aren't related in any way. Why not just forget all symmetry if this is
an issue? But is it really a bad thing if by searching you find that
there's a chr for both str and bytes?

If I think, "I want to turn my int into a bytes 'character' kind of in
the way that chr turns my int into a str". What am I going to search
or google for? I can't speak for others, but I would probably search
for something that contains 'chr' and 'bytes'.

Based on this, I'm unable to see the search disadvantage of bytes.chr.

[...]
>> bytes.char(or bytes.chr or bchr in builtins)
>> bytes.chars, bytearray.chars (sequence views)
>
> The views are already available via memoryview.cast if folks really
> want them, but encouraging their use in general isn't a great idea, as
> it means more function developers now need to ask themselves "What if
> someone passes me a char view rather than a normal bytes object?".

Thanks, I think this is the first real argument I hear against the
char view. In fact, I don't think people should ask themselves that
question, and just not accept bytes views as input. Would it be enough
to discourage storing and passing bytes views?

Anyway, the only error that would pass silently would be that the
passed-in object gets indexed (e.g. obj[0]) and a bytes-char comes out
instead of an int. But it would be a strange thing to do by the caller
to pass a char view into the bytes-consumer. I could imagine someone
wanting to pass a bytes view into a str-consumer. But there are no
significant silently-passing errors there. If str also gets .chars,
then it becomes even easier to support this.

-- Koos

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Koos Zevenhoven]

On Wed, Aug 31, 2016 at 12:20 AM, Guido van Rossum <gu...@python.org> wrote:
> I'm happy to present PEP 526 for your collective review:
> https://www.python.org/dev/peps/pep-0526/ (HTML)
> https://github.com/python/peps/blob/master/pep-0526.txt (source)
>
> There's also an implementation ready:
> https://github.com/ilevkivskyi/cpython/tree/pep-526
>
> I don't want to post the full text here but I encourage feedback on
> the high-order ideas, including but not limited to
>
> - Whether (given PEP 484's relative success) it's worth adding syntax
> for variable/attribute annotations.

While a large amount of Python programmers may not be interested in
type hinting local variables inside functions, I can see other
potential benefits in this.

When I start sketching a new class, I'm often tempted to write down
the names of the attributes first, before starting to implement
``__init__``. Sometimes I even write temporary comments for this
purpose. This syntax would naturally provide a way to sketch the list
of attributes. Yes, there is already __slots__, but I'm not sure that
is a good example of readability.

Also, when reading code, it may be hard to tell which (instance)
attributes the class implements. To have these listed in the beginning
of the class could therefore improve the readability.

In this light, I'm not sure it's a good idea to allow attribute type
hints inside methods.

>
> - Whether the keyword-free syntax idea proposed here is best:
>   NAME: TYPE
>   TARGET: TYPE = VALUE
>

I wonder if this would be better:

def NAME: TYPE
def NAME: TYPE = VALUE

Maybe it's just me, but I've always thought 'def' is Python's least
logically used keyword. It seems to come from 'define', but what is it
about 'define' that makes it relate to functions only. Adding an
optional 'def' for other variables might even be a tiny bit of added
consistency.

Note that we could then also have this:

def NAME

Which would, again for readability (see above), be a way to express
that "there is an instance variable called X, but no type hint for
now". I can't think of a *good* way to do this with the keyword-free
version for people that don't use type hints.

And then there could also be a simple decorator like
@slotted_attributes that automatically generates "__slots__" from the
annotations.

-- Koos


> Note that there's an extensive list of rejected ideas in the PEP;
> please be so kind to read it before posting here:
> https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-left-out-for-now
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 3:43 PM, Steven D'Aprano <st...@pearwood.info> wrote:
[...]
> [steve@ando ~]$ cat test.py
> from typing import Optional
>
> def foo(x:Optional[int])->int:
> if x is None:
> return -1
> return x + 1
>
> def bar(x:Optional[int])->int:
> y = x  # the type of y must be inferred
> if y is None:
> return y + 1
> return len(y)
>
> [steve@ando ~]$ mypy --strict-optional test.py
> test.py: note: In function "bar":
> test.py:11: error: Unsupported operand types for + (None and "int")
> test.py:12: error: Argument 1 to "len" has incompatible type "int"; expected 
> "Sized"
>
>
> foo passes the type check; bar fails.
>

That's great. While mypy has nice features, these examples have little
to do with PEP 526 as they don't have variable annotations, not even
using comments.

For some reason, pip install --upgrade mypy fails for me at the
moment, but at least mypy version 0.4.1 does not allow this:

from typing import Callable

def foo(cond: bool, bar : Callable, baz : Callable) -> float:
if cond:
x = bar() # type: int
else:
x = baz() # type: float
return x / 2

and complains that

test.py:7: error: Name 'x' already defined". Maybe someone can confirm
this with a newer version.

Here,

def foo(cond: bool) -> float:
if cond:
x = 1
else:
x = 1.5
return x / 2

you get a different error:

test.py:5: error: Incompatible types in assignment (expression has
type "float", variable has type "int")

Maybe someone can confirm this with a newer version, but IIUC this is
still the case.

>> I want a checker to check my code and, with minimal annotations, give me
>> confidence that my code is correct
>
> Don't we all.
>

I would add *with minimal restrictions on how the code is supposed to
be written* for type checking to work. It's not at all obvious that
everyone thinks that way. Hence, the "Semantics for type checking"
thread on python-ideas.

-- Koos

>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 7:43 PM, Nick Coghlan  wrote:
> On 4 September 2016 at 21:32, Ivan Levkivskyi  wrote:
>> The first form still could be interpreted by type checkers
>> as annotation for value (a cast to more precise type):
>>
>> variable = cast(annotation, value) # visually also looks similar
>
> I think a function based spelling needs to be discussed further, as it
> seems to me that at least some of the goals of the PEP could be met
> with a suitable definition of "cast" and "declare", with no syntactic
> changes to Python. Specifically, consider:
>
> def cast(value, annotation):
> return value
>

typing.cast already exists.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 12:51 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 4 September 2016 at 08:11, Random832 <random...@fastmail.com> wrote:
>> On Sat, Sep 3, 2016, at 18:06, Koos Zevenhoven wrote:
>>> I guess one reason I don't like bchr (nor chrb, really) is that they
>>> look just like a random sequence of letters in builtins, but not
>>> recognizable the way asdf would be.
>>>
>>> I guess I have one last pair of suggestions for the name of this
>>> function: bytes.chr or bytes.char.
>
> The PEP started out with a classmethod, and that proved problematic
> due to length and the expectation of API symmetry with bytearray. A
> new builtin paralleling chr avoids both of those problems.
>
>> What about byte? Like, not bytes.byte, just builtins.byte.
>
> The main problem with "byte" as a name is that "bytes" is *not* an
> iterable of these - it's an iterable of ints. That concern doesn't
> arise with chr/str as they're both abbreviated singular nouns rather
> than one being the plural form of the other (it also doesn't hurt that
> str actually is an iterable of chr results).
>

Since you agree with me about this...

[...]
>
> That said, the PEP does propose "getbyte()" and "iterbytes()" for
> bytes-oriented indexing and iteration, so there's a reasonable
> consistency argument in favour of also proposing "byte" as the builtin
> factory function:
>
> * data.getbyte(idx) would be a more efficient alternative to byte(data[idx])
> * data.iterbytes() would be a more efficient alternative to map(byte, data)
>

.. I don't understand the argument for having 'byte' in these names.
They should have 'char' or 'chr' in them for exacly the same reason
that the proposed builtin should have 'chr' in it instead of 'byte'.
If 'bytes' is an iterable of ints, then get_byte should probably
return an int

I'm sorry, but this argument comes across as "were're proposing the
wrong thing here, so for consistency, we might want to do the wrong
thing in this other part too".

And didn't someone recently propose deprecating iterability of str
(not indexing, or slicing, just iterability)? Then str would also need
a way to provide an iterable or sequence view of the characters. For
consistency, the str functionality would probably need to mimic the
approach in bytes. IOW, this PEP may in fact ultimately dictate how to
get a iterable/sequence from a str object.

-- Koos


> With bchr, those mappings aren't as clear (plus there's a potentially
> unwanted "text" connotation arising from the use of the "chr"
> abbreviation).
>

Which mappings?

> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 1:52 PM, Mark Shannon <m...@hotpy.org> wrote:
[...]
>
> The key difference is in placement.
> PEP 484 style
> variable = value # annotation
>
> Which reads to me as if the annotation refers to the value.
> PEP 526
> variable: annotation = value
>
> Which reads very much as if the annotation refers to the variable.
> That is a change in terms of semantics and a change for the worse, in terms
> of expressibility.
>

You have probably noticed this already, but in the semantics which I
have now explained more precisely on python-ideas

https://mail.python.org/pipermail/python-ideas/2016-September/042076.html

an annotation like

variable: annotation = value

is a little closer to an expression annotation. I.e. it does not say
that 'variable' should *always* have the type given by 'annotation'.

-- Koos

>
> Cheers,
> Mark.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 6:38 PM, Nick Coghlan  wrote:
>
> There are two self-consistent sets of names:
>

Let me add a few. I wonder if this is really used so much that
bytes.chr is too long to type (and you can do bchr = bytes.chr if you
want to):

bytes.chr (or bchr in builtins)
bytes.chr_at, bytearray.chr_at
bytes.iterchr, bytearray.iterchr

bytes.chr (or bchr in builtins)
bytes.chrview, bytearray.chrview (sequence views)

bytes.char(or bytes.chr or bchr in builtins)
bytes.chars, bytearray.chars (sequence views)


> bchr
> bytes.getbchr, bytearray.getbchr
> bytes.iterbchr, bytearray.iterbchr
>
> byte
> bytes.getbyte, bytearray.getbyte
> bytes.iterbytes, bytearray.iterbytes
>
> The former set emphasises the "stringiness" of this behaviour, by
> aligning with the chr() builtin
>
> The latter set emphasises that these APIs are still about working with
> arbitrary binary data rather than text, with a Python "byte"
> subsequently being a length 1 bytes object containing a single integer
> between 0 and 255, rather than "What you get when you index or iterate
> over a bytes instance".
>
> Having noticed the discrepancy, my personal preference is to go with
> the latter option (since it better fits the "executable pseudocode"
> ideal and despite my reservations about "bytes objects contain int
> objects rather than byte objects", that shouldn't be any more
> confusing in the long run than explaining that str instances are
> containers of length-1 str instances). The fact "byte" is much easier
> to pronounce than bchr (bee-cher? bee-char?) also doesn't hurt.
>
> However, I suspect we'll need to put both sets of names in front of
> Guido and ask him to just pick whichever he prefers to get it resolved
> one way or the other.
>
>> And didn't someone recently propose deprecating iterability of str
>> (not indexing, or slicing, just iterability)? Then str would also need
>> a way to provide an iterable or sequence view of the characters. For
>> consistency, the str functionality would probably need to mimic the
>> approach in bytes. IOW, this PEP may in fact ultimately dictate how to
>> get a iterable/sequence from a str object.
>
> Strings are not going to become atomic objects, no matter how many
> times people suggest it.
>

You consider all non-iterable objects atomic? If str.__iter__ raises
an exception, it does not turn str somehow atomic. I wouldn't be
surprised by breaking changes of this nature to python at some point.
The breakage will be quite significant, but easy to fix.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi <levkivs...@gmail.com> wrote:
> On 4 September 2016 at 19:59, Nick Coghlan <ncogh...@gmail.com> wrote:
[...]
>>
>> Similarly, it would be reasonable to say that these three snippets
>> should all be equivalent from a typechecking perspective:
>>
>> x = None # type: Optional[T]
>>
>> x: Optional[T] = None
>>
>> x: Optional[T]
>> x = None
>
>
> Nice idea, explicit is better than implicit.
>

How is it going to help that these are equivalent within one checker,
if the meaning may differ across checkers?

-- Koos


> --
> Ivan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Koos Zevenhoven]

On Thu, Sep 1, 2016 at 5:46 PM, Guido van Rossum <gu...@python.org> wrote:
> On Thu, Sep 1, 2016 at 6:11 AM, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>> While a large amount of Python programmers may not be interested in
>> type hinting local variables inside functions, I can see other
>> potential benefits in this.
>
> IOW, PEP 3157 is not dead yet. Indeed.
>

PEP 3157? Is that a typo or is there such a thing somewhere?

[...]
>> Also, when reading code, it may be hard to tell which (instance)
>> attributes the class implements. To have these listed in the beginning
>> of the class could therefore improve the readability.
>
> Right. That has been my observation using PEP 484's type comments
> extensively for annotating instance variables at the class level. E.g.
> much of mypy's own code is written this way, and it really is a huge
> help. But
>
> foo = None  # type: List[int]
>
> while it gives me the info I'm looking for, is not great
> notation-wise, and that's why I started thinking about an alternative:
>
> foo: List[int]
>
> (in either case, the __init__ contains something like `self.foo = []`).
>
>> In this light, I'm not sure it's a good idea to allow attribute type
>> hints inside methods.
>
> Those are meant for the coding style where all attributes are
> initialized in the method and people just want to add annotations
> there. This is already in heavy use in some PEP-484-annotated code
> bases I know of, using # type comments, and I think it will be easier
> to get people to switch to syntactic annotations if they can
> mechanically translate those uses. (In fact we are planning an
> automatic translator.)

I suppose the translator would be somewhat more complicated if it were
to move the type hints to the beginning of the class suite. Anyway, I
hope there will at least be a recommendation somewhere (PEP 8?) to not
mix the two styles of attribute annotation (beginning of class / in
method). The whole readability benefit turns against itself if there
are some non-ClassVar variables annotated outside __init__ and then
the rest somewhere in __init__ and in whatever initialization helper
methods __init__ happens to call.

[...]
>> Note that we could then also have this:
>>
>> def NAME
>>
>> Which would, again for readability (see above), be a way to express
>> that "there is an instance variable called X, but no type hint for
>> now". I can't think of a *good* way to do this with the keyword-free
>> version for people that don't use type hints.
>>
>> And then there could also be a simple decorator like
>> @slotted_attributes that automatically generates "__slots__" from the
>> annotations.
>
> This I like, or something like it. It can be a follow-up design. (I.e.
> a separate PEP, once we have experiece with PEP 526.)

I think there should be a syntax for this that does not involve type
hints, but I can't seem to come up with anything that works with the
keyword-free version :(.

-- Koos


> --
> --Guido van Rossum (python.org/~guido)

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Koos Zevenhoven]

On Wed, Mar 1, 2017 at 7:42 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
> Short version:
>
> - there are some reasonable requests for async variants of contextlib APIs
> for 3.7
> - prompted by Raymond, I'm thinking it actually makes more sense to add
> these in a new `asyncio.contextlib` module than it does to add them directly
> to the existing module
> - would anyone object strongly to my asking authors of the affected PRs to
> take their changes in that direction?
>

Related to this, here's a post from two years ago in attempt to tackle
the cause of this problem (of needing async and non-async variants)
and solve it in the long term.

https://mail.python.org/pipermail/python-ideas/2015-May/033267.html

You can read the details in that thread, but in short, the idea is
that all functionality that may have to wait for something (IO etc.)
should be explicitly awaited, regardless of whether the code takes
advantage of concurrency or not. This solution is an attempt to do
this without enforcing a specific async framework.

In the post, I made up the terms "Y end" and "L end", because I did
not know what to call them. This was when the draft PEP492 was being
discussed.

L is the end that 'drives' the (chain of) coroutines, usually an event
loop. Y is the other end, the most inner co-routine in the
calling/awaiting chain that does the yields. The L and Y end together
could hide the need of two variants, as explained in the above link.

—Koos

> Longer version:
>
> There are a couple of open issues requesting async variants of some
> contextlib APIs (asynccontextmanager and AsyncExitStack). I'm inclined to
> accept both of them, but Raymond raised a good question regarding our
> general design philosophy for these kinds of additions: would it make more
> sense to put these in an "asyncio.contextlib" module than it would to add
> them directly to contextlib itself?
>
> The main advantage I see to the idea is that if someone proposed adding an
> "asyncio" dependency to contextlib, I'd say no. For the existing
> asynccontextmanager PR, I even said no to adding that dependency to the
> standard contextlib test suite, and instead asked that the new tests be
> moved out to a separate file, so the existing tests could continue to run
> even if asyncio was unavailable for some reason.
>
> While rejecting the idea of an asyncio dependency isn't a problem for
> asyncontextmanager specifically (it's low level enough for it not to
> matter), it's likely to be more of a concern for the AsyncExitStack API,
> where the "asyncio.iscoroutinefunction" introspection API is likely to be
> quite helpful, as are other APIs like `asyncio.ensure_future()`.
>
> So would folks be OK with my asking the author of the PR for
> https://bugs.python.org/issue29679 (adding asynccontextmanager) to rewrite
> the patch to add it as asyncio.contextlib.asyncontextmanager (with a
> cross-reference from the synchronous contextlib docs), rather than the
> current approach of adding it directly to contextlib?
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for...else

[Koos Zevenhoven]

On Mon, Jul 24, 2017 at 7:14 PM, Steven D'Aprano <st...@pearwood.info>
wrote:

> Hello Kiuhnm, and welcome.
>
> On Mon, Jul 24, 2017 at 05:35:03PM +0200, Kiuhnm via Python-Dev wrote:
> > Hello,
> >
> > I think that the expression "for...else" or "while...else" is completely
> > counter-intuitive.
>
>
> You may be right -- this has been discussed many, many times before. In
> my personal opinion, the best (and only accurate!) phrase would have
> been:
>
> for item in sequence:
> # block
> then:
> # block
>
>
​IMO,

for item in sequence:
# block
nobreak:   # or perhaps `if not break:`
# block

would be clearer (if the syntax is necessary at all).


​[...]


> ​
>
> > Wouldn't it be possible to make it clearer? Maybe
> > something like
>
> At this point, no, it is not practical to change the syntax used. Maybe
> when Python 3.0 was first introduced, but that ship has long sailed. It
> is very, very unlikely that the syntax for this will ever change, but if
> it does, it probably won't be until something in the distant future like
> Python 5.
>

​I don't have a strong opinion on this particular case, but if something
like this is changed in Python 5, I think the decision should be made much
earlier (now?) so that the old else syntax could be discouraged (and new
syntax potentially already introduced). The same thing would apply to many
other "possibly in Python 5" changes, where there is no reason to expect
that the situation is somehow different years later.

-- Koos


>
> But not Python 4: Guido has already ruled that Python 4 will not include
> major backwards-incompatible changes. Going from 3 to 4 will not be as
> disruptive as going from 2 to 3.
>
>
​[...]​


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] for...else

[Koos Zevenhoven]

On Jul 27, 2017 02:38, "MRAB" <pyt...@mrabarnett.plus.com> wrote:

On 2017-07-26 23:55, Koos Zevenhoven wrote:

>
> ​IMO,
>
> for item in sequence:
>  # block
> nobreak:   # or perhaps `if not break:`
>  # block
>
> would be clearer (if the syntax is necessary at all).
>

You couldn't have "if not break:" because that would look like the start of
an 'if' statement.


Do you mean as an implementation issue or for human readability?

"nobreak" would introduce a new keyword, but "not break" wouldn't.


Sure :)

-- Koos (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?

[Koos Zevenhoven]

On Thu, May 4, 2017 at 1:07 AM, Terry Reedy <tjre...@udel.edu> wrote:
> On 5/3/2017 2:15 PM, Brett Cannon wrote:
>>
>> My allergies have hit me hard so I'm not thinking at full capacity, but
>> did we ever decide if supporting os.PathLike in the stdlib was viewed as an
>> enhancement or bugfix? Specifically I'm thinking of
>> https://bugs.python.org/issue30218 for adding support to
>> shutil.unpack_archive() and whether it should be backported to 3.6.
>
>
> On the face of it, that particular issue looks like an enhancement that
> should have gone into 3.6

Agreed.

> (if ever), but did not.  I notice that
> https://www.python.org/dev/peps/pep-0519/#implementation
> did not include "Update shutil", so it was not done, at least not
> completely.

Shutil was among the most important to be updated, IMO.

I had made some sort of list of affected modules elsewhere [1]:
ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to
say os.DirEntry, but that is
not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile,
tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob
and fnmatch (are the patterns paths?)

It looks like what made it to PEP519 was mainly this:

"It is expected that most APIs in Python's standard library that
currently accept a file system path will be updated appropriately to
accept path objects (whether that requires code or simply an update to
documentation will vary)."

> Was shutil updated at all?  Is unpack_archive the only shutil function not
> updated?  If so, I could see the omission as a bug.
>
> If the patch for 30218 were applied in 3.6, would the doc
> https://docs.python.org/3/library/shutil.html#shutil.unpack_archive
> need to be changed, with a note "Added in 3.6.2: filename can be any
> pathlike object"?  If so, it is an enhancement.

Regardless of bugfix vs enhancement semantics, that seems like a good
thing to do.

-- Koos

[1] e.g. in this thread somewhere:
https://mail.python.org/pipermail/python-ideas/2016-April/039827.html


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 484 update proposal: annotating decorated declarations

[Koos Zevenhoven]

On Tue, May 9, 2017 at 8:19 PM, Guido van Rossum <gu...@python.org> wrote:
> There's a PR to the peps proposal here:
> https://github.com/python/peps/pull/242
>
> The full text of the current proposal is below. The motivation for this is
> that for complex decorators, even if the type checker can figure out what's
> going on (by taking the signature of the decorator into account), it's
> sometimes helpful to the human reader of the code to be reminded of the type
> after applying the decorators (or a stack thereof). Much discussion can be
> found in the PR. Note that we ended up having `Callable` in the type because
> there's no rule that says a decorator returns a function type (e.g.
> `property` doesn't).
>
> This is a small thing but I'd like to run it by a larger audience than the
> core mypy devs who have commented so far. There was a brief discussion on
> python-ideas (my original, favorable reply by Nick, my response).
>
> Credit for the proposal goes to Naomi Seyfer, with discussion by Ivan
> Levkivskyi and Jukka Lehtosalo.
>
> If there's no further debate here I'll merge it into the PEP and an
> implementation will hopefully appear in the next version of the typing
> module (also hopefully to be included in CPython 3.6.2 and 3.5.4).
>

So the change would only affect early adopters of this typing feature,
who are likely to upgrade to newer python versions often? Could this
be called a 3.7 feature with a clearly documented bonus that it also
works in 3.6.2+ and 3.5.4+? I mean, to prevent 3rd-party libraries
tested with 3.5(.4) from being broken in 3.5.3?

> Here's the proposed text (wordsmithing suggestions in the PR please):
>
> +Decorators
> +--
> +
> +Decorators can modify the types of the functions or classes they
> +decorate. Use the ``decorated_type`` decorator to declare the type of
> +the resulting item after all other decorators have been applied::
> +
> + from typing import ContextManager, Iterator, decorated_type
> + from contextlib import contextmanager
> +
> + class DatabaseSession: ...
> +
> + @decorated_type(Callable[[str], ContextManager[DatabaseSession]])
> + @contextmanager
> + def session(url: str) -> Iterator[DatabaseSession]:
> + s = DatabaseSession(url)
> + try:
> + yield s
> + finally:
> + s.close()
> +
> +The argument of ``decorated_type`` is a type annotation on the name
> +being declared (``session``, in the example above). If you have
> +multiple decorators, ``decorated_type`` must be topmost. The
> +``decorated_type`` decorator is invalid on a function declaration that
> +is also decorated with ``overload``, but you can annotate the
> +implementation of the overload series with ``decorated_type``.
> +
>

Would __annotations__ be set by the decorator? To me, not setting them
would seem weird, but cases where the result is not a function could
be weird. I also don't see a mention of this only working in stubs.

I like Jukka's version, as it has a clear distinction between
functions and other attributes. But that might require a language
change to provide __annotations__ in a clean manner? Maybe that
language change would be useful elsewhere.

—Koos



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 484 update proposal: annotating decorated declarations

[Koos Zevenhoven]

On Fri, Jun 2, 2017 at 6:34 PM, Naomi Seyfer <na...@seyfer.org> wrote:
> Yep, interested in implementing it!  I will put implementation time on my
> schedule and tell y'all when it is, for holding myself accountable -- it
> turns out I never do anything not on my schedule.
>

I still don't understand what would happen with __annotations__. If
the decorator returns a non-function, one would expect the annotations
to be in the __annotations__ attribute of the enclosing class or
module. If it returns a function, they would be in the __annotations__
attribute of the function. And I'm talking about the runtime behavior
in Python as explained in PEP484 and PEP526. I would expect these
declarations to behave according to the same principles as other ways
to annotate variables/functions. If there is no runtime behavior, a
comment-based syntax might be more appropriate. Or have I missed
something?


—Koos



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 484 update proposal: annotating decorated declarations

[Koos Zevenhoven]

On Fri, Jun 2, 2017 at 8:57 PM, Guido van Rossum <gu...@python.org> wrote:
> On Fri, Jun 2, 2017 at 9:41 AM, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>>
>> I still don't understand what would happen with __annotations__. If
>> the decorator returns a non-function, one would expect the annotations
>> to be in the __annotations__ attribute of the enclosing class or
>> module. If it returns a function, they would be in the __annotations__
>> attribute of the function. And I'm talking about the runtime behavior
>> in Python as explained in PEP484 and PEP526. I would expect these
>> declarations to behave according to the same principles as other ways
>> to annotate variables/functions. If there is no runtime behavior, a
>> comment-based syntax might be more appropriate. Or have I missed
>> something?
>
>
> So when returning a function, the runtime version of the decorator can
> easily update the function's __annotations__. But when returning a
> non-function, the decorator would have a hard time updating
__annotations__
> of the containing class/module without "cheating" (e.g. sys._getframe()).
I
> think the latter is similar to e.g. attributes defined with @property --
> those don't end up in __annotations__ either. I think this is an
acceptable
> deficiency.
>

I suppose it is, especially because there seems to be nothing that prevents
you from getting runtime annotations in the enclosing class/module
​:


number: int

@call
def number():
return 42


But for functions one could have (
​using
 the context manager example):


def session(url: str) -> ContextManager[DatabaseSession]: ...

@predeclared
@contextmanager
def session(url: str) -> Iterator[DatabaseSession]:
s = DatabaseSession(url)
try:
yield s
finally:
s.close()


This makes it clear that the function is declared elsewhere. But the
`predeclared` decorator would need tricks like sys._getframe(1) to set
session.__annotations__ according to the predeclaration.

-- Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?

[Koos Zevenhoven]

On May 5, 2017 10:39 PM, "Chris Barker"  wrote:

Sorry to come late to the game, It wasn't immediately clear to me what the
implications were of the "enhancement or bugfix" distinction...

On Thu, May 4, 2017 at 9:46 PM, Nick Coghlan  wrote:

> That improved casting mechanism and the implicit support in the low
> level APIs is the main benefit I see in PEP 519, and if we were
> talking about an os module API that was missing os.path support, I'd
> be more likely to be on the "it's a bug" side of things.
>

absolutely.


> It's only higher level APIs like shutil that I put into the same
> category as third party libraries for now: Python 3.6 users shouldn't
> expect implicit use of the fspath protocol to be universal yet,


I think stdlib packages, like shutil are very much distinct from third
party  libs, and users certainly expect the entire stdlib to be updated to
support new language features.

We very often make the distinction between third party libs and stdlibs --
in fact, that is one reason we are reluctant to add new packages to the
stdlib...

Indeed, IIRC, that was the entire motivation for PEP 519 -- it started as a
"let's make Path objects work with the stdlib", but when we looked at how
to do that, it became clear that new protocol was teh best way to do that
and also provide flexibility to do that.

And the PEP states:

"""
Changes to Python's standard library are also proposed to utilize this
protocol where appropriate to facilitate the use of path objects where
historically only str and/or bytes file system paths are accepted. The goal
is to facilitate the migration of users towards rich path objects while
providing an easy way to work with code expecting str or bytes .
"""

It doesn't actually say "everywhere possible in the stdlib", but if the
goal is to facilitate migration, as stated, then the any but truly obscure
functions should be covered -- and shutil is certainly not obscure.


Indeed.

So it would be really great if any updates to shutils (and other stdlib
packages) to support the new protocol be back-ported.

Yes, it's "just" adding an extra call, but in my experience, low barrier to
entry are enough to discourage adoption -- and handful of shutil function
failing will certainly be enough to keep some folks from adopting the new
approach.


I suppose the worst thing to happen is what Eric describes. But technically
speaking, passing os.PathLike to shutil functions might currently be
undefined behavior, so the change is 'legal' if it's not documented ;).

-- Koos (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?

[Koos Zevenhoven]

On Thu, May 4, 2017 at 8:30 PM, Terry Reedy <tjre...@udel.edu> wrote:
> On 5/4/2017 10:43 AM, Koos Zevenhoven wrote:
>> On Thu, May 4, 2017 at 4:19 AM, Terry Reedy <tjre...@udel.edu> wrote:
>>> Enhancing public APIs in normal (non-provisional) modules in bugfix
>>> releases
>>> has turned out to be a bad thing to do.  Hence the policy to not do that.
>>> The few exceptions have been necessary to fix a bug that needed to be
>>> fixed,
>>> and could not reasonably be fixed otherwise.
>>
>> Such exceptions can of course more easily be made when the adoption of
>> a version is still small, and almost all users will never see X.Y.0 or
>> X.Y.1.
>
> This is not an allowed excuse for breaking the policy.  The x.y language is
> defined when x.y.0 is released.  Please stop.
>

Don't worry, I didn't even start :)

I do think it can cause problems if most of a stdlib module supports
PathLike and some parts do not. People will write code in 3.7
believing it works on 3.6, while it doesn't. Anyway, I'm completely
happy if the policy outweighs this issue, and I have absolutely no
need to argue about the decision.

—Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?

[Koos Zevenhoven]

On Thu, May 4, 2017 at 4:19 AM, Terry Reedy <tjre...@udel.edu> wrote:
> On 5/3/2017 7:13 PM, Koos Zevenhoven wrote:
>>
[...]

>> Shutil was among the most important to be updated, IMO.
>>
>> I had made some sort of list of affected modules elsewhere [1]:
>> ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to
>> say os.DirEntry, but that is
>> not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile,
>> tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob
>> and fnmatch (are the patterns paths?)
>
>
> What did not get done for 3.6 should be proposed for 3.7.
>

Anyone, feel free. The nightmare part is done, so this could be a case
where a PR actually pays off in terms of being able to use the
feature. There's no need for any unnecessary masochism (should there
ever be?).

[...]
>
> Enhancing public APIs in normal (non-provisional) modules in bugfix releases
> has turned out to be a bad thing to do.  Hence the policy to not do that.
> The few exceptions have been necessary to fix a bug that needed to be fixed,
> and could not reasonably be fixed otherwise.

Such exceptions can of course more easily be made when the adoption of
a version is still small, and almost all users will never see X.Y.0 or
X.Y.1. The fraction of 3.6 users is probably super tiny right now, and
even those users are likely to eagerly update to bugfix releases. For
instance, are there any major (LTS?) linux distros that already come
with 3.6.0 or 3.6.1? Well OK, 3.6.2 may be too late for some.

—Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v3 (new interpreters module)

[Koos Zevenhoven]

IV-like proxy
>>   + you wrap the object, send() the proxy, and recv() a proxy
>>   + this is entirely compatible with tp_share()
>>
>
> * Allow for multiple channel types, such that MemChannel is merely the
> *first* channel type, rather than the *only* channel type
>   + Allows PEP 554 to be restricted to things we already know can be made
> to work
>   + Doesn't block the introduction of an object-sharing based Channel in
> some future release
>   + Allows for at least some channel types to be adapted for use with
> shared memory and multiprocessing
>
>
>> Here are what I consider the key metrics relative to the utility of a
>> solution (not in any significant order):
>>
>> * how hard to understand as a Python programmer?
>>
>
> Not especially important yet - this is more a criterion for the final API,
> not the initial experimental platform.
>
>
>> * how much extra work (if any) for folks calling Channel.send()?
>> * how much extra work (if any) for folks calling Channel.recv()?
>>
>
> I don't think either are particularly important yet, although we also
> don't want to raise any pointless barriers to experimentation.
>
>
>> * how complex is the CPython implementation?
>>
>
> This is critical, since we want to minimise any potential for undesirable
> side effects on regular single interpreter code.
>
>
>> * how hard to understand as a type author (wanting to add support for
>> their type)?
>> * how hard to add support for a new type?
>> * what variety of types could be supported?
>> * what breadth of experimentation opens up?
>>
>
> You missed the big one: what risk does the initial channel design pose to
> the underlying objective of making the GIL a genuinely per-interpreter lock?
>
> If we don't eventually reach the latter goal, then subinterpreters won't
> really offer much in the way of compelling benefits over just using a
> thread pool and queue.Queue.
>
> MemChannel poses zero additional risk to that, since we wouldn't be
> sharing actual Python objects between interpreters, only C pointers and
> structs.
>
> By contrast, introducing an object channel early poses significant new
> risks to that goal, since it will force you to solve hard protocol design
> and refcount management problems *before* making the switch, rather than
> being able to defer the design of the object channel protocol until *after*
> you've already enabled the ability to run subinterpreters in completely
> independent threads.
>
>
>> The most important thing to me is keeping things simple for Python
>> programmers.  After that is ease-of-use for type authors.  However, I
>> also want to put us in a good position in 3.7 to experiment
>> extensively with subinterpreters, so that's a big consideration.
>>
>> Consequently, for PEP 554 my goal is to find a solution for object
>> sharing that keeps things simple in Python while laying a basic
>> foundation we can build on at the C level, so we don't get locked in
>> but still maximize our opportunities to experiment. :)
>>
>
> I think our priorities are quite different then, as I believe PEP 554
> should be focused on defining a relatively easy to implement API that
> nevertheless makes it possible to write interesting programs while working
> on the goal of making the GIL per-interpreter, without worrying too much
> about whether or not the initial cross-interpreter communication channels
> closely resemble the final ones that will be intended for more general use.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> k7hoven%40gmail.com
>
>


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating time for `import requests`

[Koos Zevenhoven]

On Sun, Oct 8, 2017 at 11:02 AM, David Cournapeau <courn...@gmail.com>
wrote:

>
> On Mon, Oct 2, 2017 at 6:42 PM, Raymond Hettinger <
> raymond.hettin...@gmail.com> wrote:
>
>>
>> > On Oct 2, 2017, at 12:39 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
>> >
>> >  "What requests uses" can identify a useful set of
>> > avoidable imports. A Flask "Hello world" app could likely provide
>> > another such sample, as could some example data analysis notebooks).
>>
>> Right.  It is probably worthwhile to identify which parts of the library
>> are typically imported but are not ever used.  And likewise, identify a
>> core set of commonly used tools that are going to be almost unavoidable in
>> sufficiently interesting applications (like using requests to access a REST
>> API, running a micro-webframework, or invoking mercurial).
>>
>> Presumably, if any of this is going to make a difference to end users, we
>> need to see if there is any avoidable work that takes a significant
>> fraction of the total time from invocation through the point where the user
>> first sees meaningful output.  That would include loading from nonvolatile
>> storage, executing the various imports, and doing the actual application.
>>
>> I don't expect to find anything that would help users of Django, Flask,
>> and Bottle since those are typically long-running apps where we value
>> response time more than startup time.
>>
>> For scripts using the requests module, there will be some fruit because
>> not everything that is imported is used.  However, that may not be
>> significant because scripts using requests tend to be I/O bound.  In the
>> timings below, 6% of the running time is used to load and run python.exe,
>> another 16% is used to import requests, and the remaining 78% is devoted to
>> the actual task of running a simple REST API query. It would be interesting
>> to see how much of the 16% could be avoided without major alterations to
>> requests, to urllib3, and to the standard library.
>>
>
> It is certainly true that for a CLI tool that actually makes any network
> I/O, especially SSL, import times will quickly be negligible. It becomes
> tricky for complex tools, because of error management. For example, a
> common pattern I have used in the past is to have a high level "catch all
> exceptions" function that dispatch the CLI command:
>
> try:
> main_function(...)
> except ErrorKind1:
> 
> except requests.exceptions.SSLError:
> # gives complete message about options when receiving SSL errors, e.g.
> invalid certificate
>
> This pattern requires importing requests every time the command is run,
> even if no network IO is actually done. For complex CLI tools, maybe most
> command don't use network IO (the tool in question was a complete packages
> manager), but you pay ~100 ms because of requests import for every command.
> It is particularly visible because commands latency starts to be felt
> around 100-150 ms, and while you can do a lot in python in 100-150 ms, you
> can't do much in 0-50 ms.
>
>
Yes. ​OTOH, ​it can also happen that the *imports* are in fact what use the
network IO. At the office, I usually import from a network drive. For
instance, `import requests` takes a little less than a second, and `import
IPython` usually takes more than a second, with some variation.

––Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating time for `import requests`

[Koos Zevenhoven]

On Sun, Oct 8, 2017 at 2:44 PM, Chris Angelico <ros...@gmail.com> wrote:

> On Sun, Oct 8, 2017 at 7:02 PM, David Cournapeau <courn...@gmail.com>
> wrote:
> > It is certainly true that for a CLI tool that actually makes any network
> > I/O, especially SSL, import times will quickly be negligible. It becomes
> > tricky for complex tools, because of error management. For example, a
> common
> > pattern I have used in the past is to have a high level "catch all
> > exceptions" function that dispatch the CLI command:
> >
> > try:
> > main_function(...)
> > except ErrorKind1:
> > 
> > except requests.exceptions.SSLError:
> > # gives complete message about options when receiving SSL errors,
> e.g.
> > invalid certificate
> >
> > This pattern requires importing requests every time the command is run,
> even
> > if no network IO is actually done. For complex CLI tools, maybe most
> command
> > don't use network IO (the tool in question was a complete packages
> manager),
> > but you pay ~100 ms because of requests import for every command. It is
> > particularly visible because commands latency starts to be felt around
> > 100-150 ms, and while you can do a lot in python in 100-150 ms, you
> can't do
> > much in 0-50 ms.
>
> This would be a perfect use-case for lazy importing, then. You'd pay
> the price of the import only if you get an error that isn't caught by
> one of the preceding except blocks.
>


​I suppose it might be convenient to be able to do something like:

with autoimport:
try:
main_function(...)
   ​ except ErrorKind1:
...
except requests.exceptions.SLLError:
...


The easiest workaround at the moment is still pretty clumsy:

def import_SLLError():
from requests.exceptions import SLLError
return SLLError

...


except import_SLLError():


But what happens if that gives you an ImportError?

––Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)

[Koos Zevenhoven]

On Oct 3, 2017 01:11, "Koos Zevenhoven" <k7ho...@gmail.com> wrote:

On Oct 3, 2017 01:00, "Guido van Rossum" <gu...@python.org> wrote:

 Mon, Oct 2, 2017 at 2:52 PM, Koos Zevenhoven <k7ho...@gmail.com> wrote

I don't mind this (or Nathaniel ;-) being academic. The backwards
> incompatibility issue I've just described applies to any extension via
> composition, if the underlying type/protocol grows new members (like the CM
> protocol would have gained __suspend__ and __resume__ in PEP521).
>

Since you seem to have a good grasp on this issue, does PEP 550 suffer from
the same problem? (Or PEP 555, for that matter? :-)



Neither has this particular issue, because they don't extend an existing
protocol. If this thread has any significance, it will most likely be
elsewhere.


That said, I did come across this thought while trying to find flaws in my
own PEP ;)

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)

[Koos Zevenhoven]

On Oct 3, 2017 01:00, "Guido van Rossum" <gu...@python.org> wrote:

 Mon, Oct 2, 2017 at 2:52 PM, Koos Zevenhoven <k7ho...@gmail.com> wrote

I don't mind this (or Nathaniel ;-) being academic. The backwards
> incompatibility issue I've just described applies to any extension via
> composition, if the underlying type/protocol grows new members (like the CM
> protocol would have gained __suspend__ and __resume__ in PEP521).
>

Since you seem to have a good grasp on this issue, does PEP 550 suffer from
the same problem? (Or PEP 555, for that matter? :-)



Neither has this particular issue, because they don't extend an existing
protocol. If this thread has any significance, it will most likely be
elsewhere.

-- Koos (mobile)


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)

[Koos Zevenhoven]

On Oct 3, 2017 00:02, "Guido van Rossum" <gu...@python.org> wrote:

On Mon, Oct 2, 2017 at 10:13 AM, Koos Zevenhoven <k7ho...@gmail.com> wrote:

> Hi all, It was suggested that I start a new thread, because the other
> thread drifted away from its original topic. So here, in case someone is
> interested:
>
> On Oct 2, 2017 17:03, "Koos Zevenhoven <k7ho...@gmail.com> wrote:
>
> On Mon, Oct 2, 2017 at 6:42 AM, Guido van Rossum <gu...@python.org> wrote:
>
> On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven <k7ho...@gmail.com> wrote:
>
> On Oct 1, 2017 19:26, "Guido van Rossum" <gu...@python.org> wrote:
>
> Your PEP is currently incomplete. If you don't finish it, it is not even a
> contender. But TBH it's not my favorite anyway, so you could also just
> withdraw it.
>
>
> I can withdraw it if you ask me to, but I don't want to withdraw it
> without any reason. I haven't changed my mind about the big picture. OTOH,
> PEP 521 is elegant and could be used to implement PEP 555, but 521 is
> almost certainly less performant and has some problems regarding context
> manager wrappers that use composition instead of inheritance.
>
>
> It is my understanding that PEP 521 (which proposes to add optional
> __suspend__ and __resume__ methods to the context manager protocol, to be
> called whenever a frame is suspended or resumed inside a `with` block) is
> no longer a contender because it would be way too slow. I haven't read it
> recently or thought about it, so I don't know what the second issue you
> mention is about (though it's presumably about the `yield` in a context
> manager implemented using a generator decorated with
> `@contextlib.contextmanager`).
>
>
> ​Well, it's not completely unrelated to that. The problem I'm talking
> about is perhaps most easily seen from a simple context manager wrapper
> that uses composition instead of inheritance:
>
> class Wrapper:
> def __init__(self):
> self._wrapped = SomeContextManager()
>
> def __enter__(self):
> print("Entering context")
> return self._wrapped.__enter__()
>
> def __exit__(self):
> self._wrapped.__exit__()
> print("Exited context")
>
>
> Now, if the wrapped contextmanager becomes a PEP 521 one with __suspend__
> and __resume__, the Wrapper class is broken, because it does not respect
> __suspend__ and __resume__. So actually this is a backwards compatiblity
> issue.
>
>
Why is it backwards incompatible? I'd think that without PEP 521 it would
be broken in exactly the same way because there's no __suspend__/__resume__
at all.



The wrapper is (would be) broken because it depends on the internal
implementation of the wrapped CM.

Maybe the author of SomeContextManager wants to upgrade the CM to also work
in coroutines and generators. But it could be a more subtle change in the
CM implementation.

The problem becomes more serious and more obvious if you don't know which
context manager you are wrapping:

class Wrapper:
def __init__(self, contextmanager):
self._wrapped = contextmanager

def __enter__(self):
print("Entering context")
return self._wrapped.__enter__()

def __exit__(self):
self._wrapped.__exit__()
print("Exited context")


The wrapper is (would be) broken because it does not work for all CMs
anymore.


But if the wrapper is made using inheritance, the problem goes away:
>
>
> class Wrapper(SomeContextManager):
> def __enter__(self):
> print("Entering context")
> return super().__enter__()
>
> def __exit__(self):
> super().__exit__()
> print("Exited context")
>
>
> Now the wrapper cleanly inherits the new optional __suspend__ and
> __resume__ from the wrapped context manager type.
>
>
In any case this is completely academic because PEP 521 is not going to
happen. Nathaniel himself has said so (I think in the context of discussing
PEP 550).


I don't mind this (or Nathaniel ;-) being academic. The backwards
incompatibility issue I've just described applies to any extension via
composition, if the underlying type/protocol grows new members (like the CM
protocol would have gained __suspend__ and __resume__ in PEP521).


-- Koos (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)

[Koos Zevenhoven]

On Wed, Oct 4, 2017 at 3:33 PM, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 4 October 2017 at 20:22, Koos Zevenhoven <k7ho...@gmail.com> wrote:
> > On Wed, Oct 4, 2017 at 8:07 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
> >>
> >> On 3 October 2017 at 03:13, Koos Zevenhoven <k7ho...@gmail.com> wrote:
> >> > Well, it's not completely unrelated to that. The problem I'm talking
> >> > about
> >> > is perhaps most easily seen from a simple context manager wrapper that
> >> > uses
> >> > composition instead of inheritance:
> >> >
> >> > class Wrapper:
> >> > def __init__(self):
> >> > self._wrapped = SomeContextManager()
> >> >
> >> > def __enter__(self):
> >> > print("Entering context")
> >> > return self._wrapped.__enter__()
> >> >
> >> > def __exit__(self):
> >> > self._wrapped.__exit__()
> >> > print("Exited context")
> >> >
> >> >
> >> > Now, if the wrapped contextmanager becomes a PEP 521 one with
> >> > __suspend__
> >> > and __resume__, the Wrapper class is broken, because it does not
> respect
> >> > __suspend__ and __resume__. So actually this is a backwards
> compatiblity
> >> > issue.
> >>
> >> This is a known problem, and one of the main reasons that having a
> >> truly transparent object proxy like
> >> https://wrapt.readthedocs.io/en/latest/wrappers.html#object-proxy as
> >> part of the standard library would be highly desirable.
> >>
> >
> > This is barely related to the problem I describe. The wrapper is not
> > supposed to pretend to *be* the underlying object. It's just supposed to
> > extend its functionality.
>
> If a wrapper *isn't* trying to act as a transparent object proxy, and
> is instead adapting it to a particular protocol, then yes, you'll need
> to update the wrapper when the protocol is extended.
>
>
​Yes, but it still means that the change in the dependency (in this case a
standard Python protocol) breaks the wrapper code.​

Remember that the wrappeR class and the wrappeD class can be implemented in
different libraries.



> That's not a backwards compatibility problem, because the only way to
> encounter it is to update your code to rely on the new extended
> protocol - your *existing* code will continue to work fine, since
> that, by definition, can't be relying on the new protocol extension.
>
>
​No, not all code is "your" code. Clearly this is not a well-known problem.
This is a backwards-compatibility problem for the author of the wrappeR,
not for the author of the wrappeD object.

––Koos
​

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)

[Koos Zevenhoven]

On Wed, Oct 4, 2017 at 4:04 PM, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 4 October 2017 at 22:45, Koos Zevenhoven <k7ho...@gmail.com> wrote:
> > On Wed, Oct 4, 2017 at 3:33 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> >> That's not a backwards compatibility problem, because the only way to
> >> encounter it is to update your code to rely on the new extended
> >> protocol - your *existing* code will continue to work fine, since
> >> that, by definition, can't be relying on the new protocol extension.
> >>
> >
> > No, not all code is "your" code. Clearly this is not a well-known
> problem.
> > This is a backwards-compatibility problem for the author of the wrappeR,
> not
> > for the author of the wrappeD object.
>
> No, you're misusing the phrase "backwards compatibility", and
> confusing it with "feature enablement".
>
> Preserving backwards compatibility just means "existing code and
> functionality don't break". It has nothing to do with whether or not
> other support libraries and frameworks might need to change in order
> to enable full access to a new language feature.
>
>
​It's not about full access to a new language feature. It's about the
wrappeR promising it can wrap any ​context manager, which it then no longer
can. If the __suspend__ and __resume__ methods are ignored, that is not
about "not having full access to a new feature" — that's broken code. The
error message you get (if any) may not contain any hint of what went wrong.

Take the length hint protocol defined in PEP 424 for example: that
> extended the iterator protocol to include a new optional
> __length_hint__ method, such that container constructors can make a
> more reasonable guess as to how much space they should pre-allocate
> when being initialised from an iterator or iterable rather than
> another container.
>
>
​This is slightly similar, but not really. Not using __length_hint__ does
not affect the correctness of code.


> That protocol means that many container wrappers break the
> optimisation. That's not a compatibility problem, it just means those
> wrappers don't support the feature, and it would potentially be a
> useful enhancement if they did.
>
>
​Again, ignoring __length_hint__ does not lead to broken code, so that just
means the wrapper is as slow or as fast as it was before.

​So I still think it's an issue for the author of the wrapper to fix––even
if just by documenting that the wrapper does not support the new protocol
members. But that would not be necessary if the wrapper uses inheritance.

(Of course there may be another reason to not use inheritance, but just
overriding two methods seems like a good case for inheritance.).
​
​This discussion seems pretty pointless by now. It's true that *some* code
needs to change for this to be a problem. Updating only the Python version
does not break a codebase if libraries aren't updated, and even then,
breakage is not very likely, I suppose.

It all depends on the kind of change that is made. For __length_hint__, you
only risk not getting the performance improvement. For __suspend__ and
__resume__, there's a small chance of problems. For some other change, it
might be even riskier. But this is definitely not the most dangerous type
of compatibility issue.

––Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)

[Koos Zevenhoven]

On Wed, Oct 4, 2017 at 8:07 AM, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 3 October 2017 at 03:13, Koos Zevenhoven <k7ho...@gmail.com> wrote:
> > Well, it's not completely unrelated to that. The problem I'm talking
> about
> > is perhaps most easily seen from a simple context manager wrapper that
> uses
> > composition instead of inheritance:
> >
> > class Wrapper:
> > def __init__(self):
> > self._wrapped = SomeContextManager()
> >
> > def __enter__(self):
> > print("Entering context")
> > return self._wrapped.__enter__()
> >
> > def __exit__(self):
> > self._wrapped.__exit__()
> > print("Exited context")
> >
> >
> > Now, if the wrapped contextmanager becomes a PEP 521 one with __suspend__
> > and __resume__, the Wrapper class is broken, because it does not respect
> > __suspend__ and __resume__. So actually this is a backwards compatiblity
> > issue.
>
> This is a known problem, and one of the main reasons that having a
> truly transparent object proxy like
> https://wrapt.readthedocs.io/en/latest/wrappers.html#object-proxy as
> part of the standard library would be highly desirable.
>
>
This is barely related to the problem I describe. The wrapper is not
supposed to pretend to *be* the underlying object. It's just supposed to
extend its functionality.

Maybe it's just me, but using a transparent object proxy for this sounds
like someone trying to avoid inheritance for no reason and at any cost.
Inheritance probably has faster method access, and makes it more obvious
what's going on:

def Wrapper(contextmanager):
class Wrapper(type(contextmanager)):
def __enter__(self):
print("Entering context")
return contextmanager.__enter__()

def __exit__(self):
contextmanager.__exit__()
print("Exited context")
return Wrapper()


A wrapper based on a transparent object proxy is just a non-transparent
replacement for inheritance. Its wrapper nature is non-transparent because
it pretends to `be` the original object, while it's actually a wrapper.

But an object cannot `be` another object as long as the `is` operator won't
return True. And any straightforward way to implement that would add
performance overhead for normal objects.

I do remember sometimes wanting a transparent object proxy. But not for
normal wrappers. But I don't think I've gone as far as looking for a
library to do that, because it seems that you can only go half way anyway.

––Koos

​--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v3 (new interpreters module)

[Koos Zevenhoven]

On Wed, Oct 4, 2017 at 4:51 PM, Eric Snow <ericsnowcurren...@gmail.com>
wrote:

> On Tue, Oct 3, 2017 at 11:36 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> > The problem relates to the fact that there aren't any memory barriers
> > around CPython's INCREF operations (they're implemented as an ordinary
> > C post-increment operation), so you can get the following scenario:
> >
> > * thread on CPU A has the sole reference (ob_refcnt=1)
> > * thread on CPU B acquires a new reference, but hasn't pushed the
> > updated ob_refcnt value back to the shared memory cache yet
> > * original thread on CPU A drops its reference, *thinks* the refcnt is
> > now zero, and deletes the object
> > * bad things now happen in CPU B as the thread running there tries to
> > use a deleted object :)
>
> I'm not clear on where we'd run into this problem with channels.
> Mirroring your scenario:
>
> * interpreter A (in thread on CPU A) INCREFs the object (the GIL is still
> held)
> * interp A sends the object to the channel
> * interp B (in thread on CPU B) receives the object from the channel
> * the new reference is held until interp B DECREFs the object
>
> From what I see, at no point do we get a refcount of 0, such that
> there would be a race on the object being deleted.
>
> ​
So what you're saying is that when Larry finishes the gilectomy,
subinterpreters will work GIL-free too?​-)

​––Koos
​

The only problem I'm aware of (it dawned on me last night), is in the
> case that the interpreter that created the object gets deleted before
> the object does.  In that case we can't pass the deletion back to the
> original interpreter.  (I don't think this problem is necessarily
> exclusive to the solution I've proposed for Bytes.)
>
> -eric
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> k7hoven%40gmail.com
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)

[Koos Zevenhoven]

 far as the dataclass interaction with `__slots__` goes, that's a
> problem largely specific to slots (and `__metaclass__` before it), in that
> they're the only characteristics of a class definition that affect how
> CPython allocates memory for the class object itself (the descriptors for
> the slots are stored as a pointer array after the class struct, rather than
> only in the class dict).
>
> Given PEP 526 variable annotations, __slots__ could potentially benefit
> from a __metaclass__ style makeover, allowing an "infer_slots=True" keyword
> argument to type.__new__ to request that the list of slots be inferred from
> __annotations__ (Slot inference would conflict with setting class level
> default values, but that's a real conflict, as you'd be trying to use the
> same name on the class object for both the slot descriptor and the default
> value)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> k7hoven%40gmail.com
>
>


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com