Re: [Python-ideas] Generator syntax hooks?

[Guido van Rossum]

On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:

> On 8 August 2017 at 09:06, Chris Barker  wrote:
> > It would be nice to have an easier access to an "slice iterator" though
> --
> > one of these days I may write up a proposal for that.
>
> An idea I've occasionally toyed with [1] is some kind of "iterview"
> that wraps around an arbitrary iterable and produces lazy itertools
> based results rather than immediate views or copies.
>
> However, my experience is also that folks are *really* accustomed to
> syntactic operations on containers producing either full live views
> (e.g. memoryview or numpy slices, range as a dynamically computed
> container), or actual copies (builtin container types). Having them
> produce consumable iterators instead then gets confusing due to the
> number of operations that will implicitly consume them (including
> simple "x in y" checks).
>
> The OP's proposal doesn't fit into that category though: rather it's
> asking about the case where we have an infinite iterator (e.g.
> itertools.count(0)), and want to drop items until they start meeting
> some condition (i.e. itertools.dropwhile) and then terminate the
> iterator as soon as another condition is no longer met (i.e.
> itertools.takewhile).
>

I don't think that's what the OP meant. The original proposal seemed to
assume that it would be somehow reasonable for the input ("integers" in the
example) to be able to see and parse the condition in the generator
expression ("1000 <= x < 10" in the example, with "x" somehow known to
be bound to the iteration value). That's at least what I think the remark
"I like mathy syntax" referred to.


> Right now, getting the "terminate when false" behaviour requires the
> use of takewhile:
>
> {itertools.takewhile(lambda x: x < 100, itertools.count(1000)}
>
> In these cases, the standard generator expression syntax is an
> attractive nuisance because it *looks* right from a mathematical
> perspective, but hides an infinite loop:
>
> {x for x in itertools.count(0) if 1000 <= x < 100}
>
> The most credible proposal to address this that I've seen is to borrow
> the "while" keyword in its "if not x: break" interpretation to get:
>
> {x for x in itertools.count(0) if 1000 <= x while x < 100}
>
> which would be compiled as equivalent to:
>
> x = set()
> for x in itertools.count(0):
> if 1000 <= x:
> set.add(x)
> if not x < 100:
> break
>
> (and similarly for all of the other comprehension variants)
>
> There aren't any technical barriers I'm aware of to implementing that,
> with the main historical objection being that instead of the
> comprehension level while clause mapping to a while loop directly the
> way the for and if clauses map to their statement level counterparts,
> it would instead map to the conditional break in the expanded
> loop-and-a-half form:
>
> while True:
> if not condition:
> break
>
> While it's taken me a long time to come around to the idea, "Make
> subtle infinite loops in mathematical code easier to avoid" *is* a
> pretty compelling user-focused justification for incurring that extra
> complexity at the language design level.
>

I haven't come around to this yet. It looks like it will make explaining
comprehensions more complex, since the translation of "while X" into "if
not X: break" feels less direct than the translations of "for x in xs" or
"if pred(x)". (In particular, your proposal seems to require more
experience with mentally translating loops and conditions into jumps --
most regulars of this forum do that for a living, but I doubt it's second
nature for the OP.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 8 August 2017 at 09:06, Chris Barker  wrote:
> It would be nice to have an easier access to an "slice iterator" though --
> one of these days I may write up a proposal for that.

An idea I've occasionally toyed with [1] is some kind of "iterview"
that wraps around an arbitrary iterable and produces lazy itertools
based results rather than immediate views or copies.

However, my experience is also that folks are *really* accustomed to
syntactic operations on containers producing either full live views
(e.g. memoryview or numpy slices, range as a dynamically computed
container), or actual copies (builtin container types). Having them
produce consumable iterators instead then gets confusing due to the
number of operations that will implicitly consume them (including
simple "x in y" checks).

The OP's proposal doesn't fit into that category though: rather it's
asking about the case where we have an infinite iterator (e.g.
itertools.count(0)), and want to drop items until they start meeting
some condition (i.e. itertools.dropwhile) and then terminate the
iterator as soon as another condition is no longer met (i.e.
itertools.takewhile).

Right now, getting the "terminate when false" behaviour requires the
use of takewhile:

{itertools.takewhile(lambda x: x < 100, itertools.count(1000)}

In these cases, the standard generator expression syntax is an
attractive nuisance because it *looks* right from a mathematical
perspective, but hides an infinite loop:

{x for x in itertools.count(0) if 1000 <= x < 100}

The most credible proposal to address this that I've seen is to borrow
the "while" keyword in its "if not x: break" interpretation to get:

{x for x in itertools.count(0) if 1000 <= x while x < 100}

which would be compiled as equivalent to:

x = set()
for x in itertools.count(0):
if 1000 <= x:
set.add(x)
if not x < 100:
break

(and similarly for all of the other comprehension variants)

There aren't any technical barriers I'm aware of to implementing that,
with the main historical objection being that instead of the
comprehension level while clause mapping to a while loop directly the
way the for and if clauses map to their statement level counterparts,
it would instead map to the conditional break in the expanded
loop-and-a-half form:

while True:
if not condition:
break

While it's taken me a long time to come around to the idea, "Make
subtle infinite loops in mathematical code easier to avoid" *is* a
pretty compelling user-focused justification for incurring that extra
complexity at the language design level.

Cheers,
Nick.

[1] https://mail.python.org/pipermail/python-ideas/2010-April/006983.html

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Pseudo methods

[Nick Coghlan]

On 7 August 2017 at 18:48, Victor Stinner  wrote:
> Ruby provides this feature. A friend who is a long term user of Rails
> complained that Rails abuses this and it's a mess in practice. So I
> dislike this idea.

Right, Python's opinionated design guidance is to clearly distinguish
between "data first" designs using methods on objects and "algorithm
first" designs using functools.singledispatch (or similar mechanisms),
since they place different constraints on how new implementations are
added, and where you should look for more information about how an
algorithm works.

Part of the intent behind this guidance is to better enable local
reasoning about a piece of code:

from my_string_utils import has_vowels

if has_vowels(input("Enter a word: ")):
print("Contains vowels!")
else:
print("Does not contain vowels!")

Here, it is clear that if we want to know more about what "has_vowels"
does, or if we want to request changes to how it works, then
"my_string_utils" is where we need to go next.

By contrast, that's significantly less clear if our string utils
module were to implicitly modify the behaviour of input() or builtin
strings:

import my_string_utils

if input("Enter a word: ").has_vowels():
print("Contains vowels!")
else:
print("Does not contain vowels!")

To analyse and investigate this code, we need to "just know" that:

- the result of "input()" doesn't normally have a "has_vowels()" method
- therefore, importing "my_string_utils" must have either replaced the
input builtin or mutated the str type
- therefore, "my_string_utils" is probably the place to go for more
information on "has_vowels"

If our import line had instead looked like "import my_string_utils,
my_other_utils", we'd have to go look at both of them to figure out
where the "has_vowels()" method might be coming from (and hope it
wasn't happening further down as a side effect of one of the modules
*they* imported).

Injecting methods rather than writing functions that dispatch on the
type of their first argument also creates new opportunities for naming
conflicts: while "my_string_utils.has_vowels" and
"your_string_utils.has_vowels" can happily coexist in the same program
without conflicts, there's only one "input" builtin, and only one
"str" builtin.

Can this level of explicitness be an obstacle at times? Yes, it can,
especially for testing and interactive use, which is why Python offers
features like wildcard imports, runtime support for monkeypatching of
user-defined types, and runtime support for dynamically replacing
builtins and module globals. However, the concerns around the
difficulties of complexity management in the face of implicit action
at a distance remain valid, so those features all fall into the
category of "supported, but not encouraged, except in specific
circumstances".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Stephen J. Turnbull]

> Soni L. writes:
 > Steven d'Aprano writes:

 > > range(1000, 100)
 > > (x for x in range(1000, 100))  # waste of time and effort

 > Actually, those have different semantics!

That's not real important.  As Stefan Behnel points out, it's simple
(and efficient) to get iterator semantics by using iter().

The big issue here is that Python is not the kind of declarative
language where (x for x in int if 1_000 ≤ x ≤ 1_000_000)[1] is
natural to write, let alone easy to implement efficiently.  Aside from
the problem of (x for x in float if 1_000 ≤ x ≤ 1_000_000) (where
the answer is "just don't do that"), I can't think of any unbounded
collections in Python that aren't iterables, except some types.

That makes Steven's criticism pretty compelling.  If you need to
design a collection's __iter__ specially to allow it to decide whether
the subset that satisfies some condition is exhausted, why not just
subclass some appropriate existing collection with a more appropriate
__iter__?

Footnotes: 
[1]  See what I did there? ;-)

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Stefan Behnel]

Soni L. schrieb am 08.08.2017 um 01:56:
> On 2017-08-07 08:35 PM, Steven D'Aprano wrote:
>> Hi Soni, and welcome!
>>
>> On Mon, Aug 07, 2017 at 04:30:05PM -0300, Soni L. wrote:
>>
>>> What if, (x for x in integers if 1000 <= x < 100), was syntax sugar
>>> for (x for x in range(1000, 100))?
>> If you want the integers from 1000 to 100, use:
>>
>> range(1000, 100)
>>
>> Don't waste your time slowing down the code with an unnecessary and
>> pointless wrapper that does nothing but pass every value on unchanged:
>>
>> (x for x in range(1000, 100))  # waste of time and effort
> 
> Actually, those have different semantics!
> 
 x = range(1, 10)
 list(x)
> [1, 2, 3, 4, 5, 6, 7, 8, 9]
 list(x)
> [1, 2, 3, 4, 5, 6, 7, 8, 9]
> 
 x = (x for x in range(1, 10))
 list(x)
> [1, 2, 3, 4, 5, 6, 7, 8, 9]
 list(x)
> []

In that case, use iter(range(1000, 100)).

range() creates an iterable, which is iterable more than once.

iter(range()) creates an iterator from that iterable, which has the
semantics that you apparently wanted.

Stefan

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/