Re: [Python-ideas] Generator syntax hooks?

[Markus Meskanen]

Though I still dop'nt think using "while" would really introduce that much
confusion -- sure, it doesn't introduce a new loop, but, as someone pointed
out earlier in this thread it really is only changing from a:

"while do"

to a

"do while"

construct -- so means pretty much the same thing.


+1 for the "while" from me too, I don't think most people would find it
confusing (supposing they don't find current [x for x in foo if ...]
confusing either), and introducing a break there is just more of a mess.

To those who say that it might get ugly if you do something like:

[x for y in foo for x in y while x != y if x + y < 100]

This still isn't even unbearable, and once it gets that hard, maybe you
should consider something else anyways.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

[...]
> result = []
> for x in sequence:
> if condition:
> result.append(expression)
> else:
> break
>
> which could be written
>
> [expression for x in sequence if condition break]
>
> It's what I thought too. Adding a `while` clause here just overly
> complicates the understanding of the comprehension. The `break` keyword is
> already easily understandable and helps to map the comprehension with the
> plain for-loop (I like this mapping for its reverse counterpart, as I often
> start with plain for-loops to rewrite them later to comprehensions when it
> makes sense).
>

having the "if condition" there seems confusing to me, particularly if you
want an if condition as a filter as well:

[expression for x in sequence if condition1 if condition2 break]

which makes me want:

[expression for x in sequence if condition1 breakif condition2]

adding another keyword is a pretty big deal though!

would it be possible to add a keyword ("breakif" in this case) that was
ONLY legal in comprehensions?

Though I still dop'nt think using "while" would really introduce that much
confusion -- sure, it doesn't introduce a new loop, but, as someone pointed
out earlier in this thread it really is only changing from a:

"while do"

to a

"do while"

construct -- so means pretty much the same thing.

I agree that scanning a code base to see if there really are many loops in
practice that could use this construct would be a good way to see if there
is any point.

And it would also be interesting to do a survey of "random" folks as to how
they would interpret such a construct -- it's pretty hard for a small group
to know what is and isn't "confusing"

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Paul Moore]

On 11 August 2017 at 05:49, Steven D'Aprano  wrote:
> On Thu, Aug 10, 2017 at 01:25:24PM -0700, Chris Barker wrote:
>> On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:
>>
>>
>> >  Also, there's a potential issue
>> > here - consider
>> >
>> > [expr for var in even_numbers() if is_odd(var) while var < 100]
>> >
>> > This is an infinite loop, even though it has a finite termination
>> > condition (var < 100), because we only test the termination condition
>> > if var is odd, which it never will be.
>
> I'm not sure why Paul thinks this is an issue. There are plenty of ways
> to accidentally write an infinite loop in a comprehension, or a for
> loop, already:

Mostly because I work in a support and maintenance environment, where
we routinely see code that *originally* made sense, but which was over
time modified in ways that break things - usually precisely because
coders who in theory understand how to write such things correctly,
end up not taking the time to fully understand the constructs they are
modifying. Of course that's wrong, but it's sadly all too common, and
for that reason I'm always wary of constructs that need thinking
through carefully to understand the implications.

Nick's original

{x for x in itertools.count(0) if 1000 <= x while x < 100}

was like that. It was *sort of* obvious that it meant "numbers between
1_000 and 1_000_000, but the interaction between "if" and "while"
wasn't clear to me. If I were asked to rush in a change to only pick
odd numbers,

{x for x in itertools.count(0) if 1000 <= x and is_odd(x) while x < 100}

seems right to me, but quick - what about edge cases? It's not that I
can't get it right, nor is it that I can't test that I *did* get it
right, just that this sort of "quick fix" is very common in the sort
of real-world coding I see regularly, and a huge advantage of Python
is that it's hard to get in a situation where the obvious guess is
wrong.

Don't get me wrong - I'm not arguing that the sky is falling. Just
that this construct isn't as easy to understand as it seems at first
(and that hard-to-understand cases appear *before* you hit the point
where it's obvious that the statement is too complex and should be
refactored.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 11 August 2017 at 15:13, Steven D'Aprano  wrote:
> On Fri, Aug 11, 2017 at 02:34:53PM +1000, Nick Coghlan wrote:
>> This is another good reason why a termination condition would need to
>> be checked before the filter condition rather than either after it, or
>> only when the filter condition was true.
>
> Why is this a problem that needs solving?

Because the most obvious interpretation of a completely independent
"while" clause in comprehensions would be as a nested loop inside the
outer for loop, not as a nested if-else-break statement.

As a result of that, I'm only personally prepared to support for-while
comprehensions if they're syntactic sugar for a combined statement
level for-while loop that makes it clear why only the "for" clauses in
a comprehension create new loops.

I *wouldn't* be prepared to support them if they could only be
explained in terms of a direct mapping to an if statement and had no
statement level counterpart that actually used the "while" keyword.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Fri, Aug 11, 2017 at 02:49:10PM +1000, Steven D'Aprano wrote:
> On Thu, Aug 10, 2017 at 01:25:24PM -0700, Chris Barker wrote:

> > I guess I think of it like this:
> > 
> > "if" is providing a filtering mechanism
> > 
> > "while" is providing a termination mechanism
> > 
> >  -- is there a use case anyone can think of when they would want the while
> > to be applied to the list AFTER filtering?


Oops, sorry I had a thinko and read your question in the opposite sense 
than it actually is. See my response to Nick for an example:

I have an iterable of arbitrary objects. I want to ignore anything that
isn't a string, and halt if the string doesn't start with "A".

[expr for s in objects if isinstance(s, str) while s.startswith("A")]


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Fri, Aug 11, 2017 at 02:34:53PM +1000, Nick Coghlan wrote:

> This is actually how I came to the conclusion that if we were ever to
> do something like this, the termination condition would need to go
> *before* the filter condition:

What if you want to check the filter condition before the termination 
condition?

I have an iterable of arbitrary objects. I want to ignore anything that 
isn't a string, and halt if the string doesn't start with "A". This is 
easy:


[expr for s in objects if isinstance(s, str) while s.startswith("A")]


Why should we prohibit expressing this, and instead write it as this?

[expr for s in objects while (s.startswith("A")) if isinstance(s, str) else 
True) if isinstance(s, str)]


Or split into multiple comprehensions?

[expr for s in [obj for obj in objects if isinstance(obj, str)] while 
s.startswith("A")]


> (expr for var in seq while loop_cond if filter_cond)
> 
> <=>
> 
> for var in seq:
> if loop_cond:
> if filter_cond:
> yield expr
> else:
> break


We can still expand the clauses if they are presented in the opposite 
order:

(expr for var in seq if filter_cond while loop_cond)

<=>

for var in seq:
if filter_cond:
if loop_cond:
yield expr
else:
break

There's no need to prohibit that. It is meaningful and useful and just 
because somebody might accidentally fail to exit an infinite loop is no 
reason to ban this.


> This is another good reason why a termination condition would need to
> be checked before the filter condition rather than either after it, or
> only when the filter condition was true.

Why is this a problem that needs solving?

Who is to say that an infinite generator expression isn't exactly what 
the programmer wants? If the halting condition is not true, the 
generator expression will either keep going until the iterator is 
exhausted, or it will be an infinite generator just like the 
unprocessed, unfiltered source iterator. This is not necessarily a 
problem.




-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 11 August 2017 at 06:53, Spencer Brown  wrote:
> The logical solution to me is to allow any order of while and if, and follow
> the same 'rule' as multiple for loops - just nest/test those in that order.
> Then you can have whatever priority you need. One question though is how
> this should handle multiple loops - break all of them, or just the current
> one?

This is why I think a for-while construct in comprehensions would
really only make sense in combination with a *statement* level
for-while construct, as the problem we have is:

- a termination condition can't readily use "if" (even in combination
with "break") because that's visually and syntactically ambiguous with
a filter condition
- a naive translation of a "while" based syntax makes it look like a
nested *non-terminating* loop

Both of those problems may be resolved if a "for-while" loop exists as
a top level looping construct that can terminate based on *either* an
iterator being exhausted *or* a condition becoming false.

The question then becomes whether or not a "for-while" loop is
actually useful enough to be added as a general construct, given that
we already have "if not condition: break" as a way of modeling a loop
ending early because a condition became false.

One way to gather evidence on that front would be to survey the
standard library for places where we use "break", and see if any of
them would be more readable given a for-while construct, whether as a
statement, or as part of the comprehension syntax. (Note: I'm not
interested enough in the idea to do that evidence gathering myself,
I'm just pointing it out in case anyone is curious enough to take the
time to collect those details)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Thu, Aug 10, 2017 at 01:25:24PM -0700, Chris Barker wrote:
> On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:
> 
> 
> >  Also, there's a potential issue
> > here - consider
> >
> > [expr for var in even_numbers() if is_odd(var) while var < 100]
> >
> > This is an infinite loop, even though it has a finite termination
> > condition (var < 100), because we only test the termination condition
> > if var is odd, which it never will be.

I'm not sure why Paul thinks this is an issue. There are plenty of ways 
to accidentally write an infinite loop in a comprehension, or a for 
loop, already:

[expr for var in even_numbers()]

will do it, if even_numbers is unexpectedly an infinite iterator. Or you 
could write:

for num in even_numbers():
if is_odd(num) and num > 100:
break

No loop syntax, whether it is functional style (takewhile, map, etc.), 
comprehension, or traditional style for loops, enables the programmer 
to avoid thinking about what they write.


> why is the termination only tested if teh if clause is True? Could then not
> be processed in parallel? or the while first

Because we're following the standard Python rule of left-to-right 
execution. The while clause is tested only if the if clause is true 
because it follows the if clause.

I think that there's an argument to be made for the rule:

We can have `if` in a comprehension, or `while`, but not both

in order to limit complexity. Analogy: 

(1) we intentionally limit the decorator @ syntax to a subset of 
expressions;

(2) likewise we intentionally allow (but don't encourage) monkey-
patching of Python classes only, not built-ins.

Just because we *can* allow arbitrary code combinations doesn't mean we 
*must*. We have a choice to say:

"No, you cannot mix `if` and `when` in the same comprehension. 
Why? Because we say so. Because it is confusing if you do."

I'd be okay with that rule.

But if we decide to allow arbitrary combinations of for/if/while in 
comprehensions, then I think we must keep the same left-to-right rule we 
have now. Currently we process multiple for/if clauses left-to-right:

[expr for x in a if cond for y in b]

is equivalent to:

for x in a:
if cond:
for y in b:
expr

rather than moving the `if` to the end. If you want it at the end, put 
it there yourself. Adding `while` shouldn't change that. It would be 
crazy-complicated to have a rule:

"the presence of a while means the comprehension is 
processed in parallel"

or

"all the while clauses are processed before (after?)
the if clauses, regardless of their order of appearance."


> so maybe better to do:
> 
> [expr for var in even_numbers() while var < 100 if is_odd(var)]

Well sure, that's the *correct* way to write the code:

for var in even_numbers():
if not (var < 100): break
if is_odd(var):
results.append(expr)

(for some definition of "correct" -- this is clearly an expensive way to 
generate an empty list.)

But in general one might wish to test the if or the while in either 
order.


> Maybe it's just me, but I would certainly expect the while to have
> precedence.

Does that apply to these idioms as well?

while cond:
if flag:
...

versus:

if flag:
while cond:
...


I would not expect them to be the same, and nor would I expect these to 
be the same:

[expr for x in seq if flag while cond]

[expr for x in seq while cond if flag]


> I guess I think of it like this:
> 
> "if" is providing a filtering mechanism
> 
> "while" is providing a termination mechanism
> 
>  -- is there a use case anyone can think of when they would want the while
> to be applied to the list AFTER filtering?

[process(n) for n in numbers while n > 0 if is_odd(n)] 

Halt on the first zero or negative number, regardless of whether it is 
even or odd, but process only odd numbers.



Paul: 
> > Obviously, this is a contrived example. And certainly "don't do 
> > that, then" is a valid response. But my instinct is that people are 
> > going to get this wrong - *especially* in a maintenance environment.

That's the argument for limiting comprehensions to either `if` or 
`while` but not both. And I actually would be okay with that -- 
especially if we leave open the possibility of relaxing the prohibition 
in the future.

But personally, I think that's under-estimating the ability of 
programmers to reason about loops. Of course a comprehension with 
multiple for/if/while clauses is hard to reason about, and we shouldn't 
*encourage* them, but we don't prohibit multiple for/if clauses. Why 
should `while` be held to a higher standard? If we allow people to shoot 
themselves in the foot by writing complex list comprehensions with ten 
`for` loops and seven `if` clauses, why should we baulk at allowing them 
a `while` clause as well?



-- 
Steve
___

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 11 August 2017 at 01:39, Paul Moore  wrote:
> On 10 August 2017 at 14:42, Steven D'Aprano  wrote:
>> I don't think it is confusing. Regardless of the implementation, the
>> meaning of:
>>
>> [expression for x in sequence while condition]
>>
>> should (I believe) be obvious to anyone who already groks comprehension
>> syntax. The mapping to a for-loop is admittedly a tad more complex:
>>
>> result = []
>> for x in sequence:
>> if not condition: break
>> result.append(expression)
>>
>> but I'm yet to meet anyone who routinely and regularly reads
>> comprehensions by converting them to for loops like that. And if they
>> did, all they need do is mentally map "while condition" to "if not
>> condition: break" and it should all Just Work™.
>
> The hard part is the interaction between if and while.
>
> Consider (expr for var in seq if cond1 while cond2):
>
> This means:
>
> for var in seq:
> if cond1:
> if not cond2: break
> yield expr
>
> Note that unlike all other comprehension clauses (for and if) while
> doesn't introduce a new level of nesting. That's an inconsistency, and
> while it's minor, it would need clarifying (my original draft of this
> email was a mess, because I misinterpreted how if and while would
> interact, precisely over this point).

This is actually how I came to the conclusion that if we were ever to
do something like this, the termination condition would need to go
*before* the filter condition:

(expr for var in seq while loop_cond if filter_cond)

<=>

for var in seq:
if loop_cond:
if filter_cond:
yield expr
else:
break

With the clauses in that order, the "while" keyword effectively
operates as "if-else-break" the same way it does in a regular while
loop, and could potentially be introduced as a modifying clause on
regular for loops at the same time.

One of the neat things the latter would allow is to make it even
easier to introduce a diagnostic loop counter into while loops:

while condition:
...

could become:

for iteration in itertools.count(1) while condition:
...

rather than having to implement a manually incremented loop counter
the way you do today.

> Also, there's a potential issue
> here - consider
>
> [expr for var in even_numbers() if is_odd(var) while var < 100]
>
> This is an infinite loop, even though it has a finite termination
> condition (var < 100), because we only test the termination condition
> if var is odd, which it never will be.

This is another good reason why a termination condition would need to
be checked before the filter condition rather than either after it, or
only when the filter condition was true.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Thu, Aug 10, 2017 at 1:53 PM, Spencer Brown  wrote:

> The logical solution to me is to allow any order of while and if, and
> follow the same 'rule' as multiple for loops - just nest/test those in that
> order.
>

Actually, I think it would be better to only allow one order, and have the
"while" always teeted first -- which may mean it should be placed first for
clarity.


> Then you can have whatever priority you need. One question though is how
> this should handle multiple loops - break all of them, or just the current
> one?
>

just the current one, just like a "break", or for that matter, a nested
while...

-CHB





> - Spencer Brown
>
> On 11 Aug 2017, at 6:27 am, Chris Barker  wrote:
>
> On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:
>
>
>>  Also, there's a potential issue
>> here - consider
>>
>> [expr for var in even_numbers() if is_odd(var) while var < 100]
>>
>> This is an infinite loop, even though it has a finite termination
>> condition (var < 100), because we only test the termination condition
>> if var is odd, which it never will be.
>>
>
> why is the termination only tested if teh if clause is True? Could then
> not be processed in parallel? or the while first
>
> so maybe better to do:
>
> [expr for var in even_numbers() while var < 100 if is_odd(var)]
>
> Maybe it's just me, but I would certainly expect the while to have
> precedence.
>
> I guess I think of it like this:
>
> "if" is providing a filtering mechanism
>
> "while" is providing a termination mechanism
>
>  -- is there a use case anyone can think of when they would want the while
> to be applied to the list AFTER filtering?
>
> Obviously, this is a contrived example. And certainly "don't do that,
>> then" is a valid response. But my instinct is that people are going to
>> get this wrong - *especially* in a maintenance environment.
>
>
> sure, but would there be an issue if teh while were given precedence?
>
> Overall, I agree with Steven's point. It seems pretty obvious what the
>> intention is, and while it's probably possible to construct examples
>> that are somewhat unclear,
>>
>> 1. The mechanical rule gives an explicit meaning
>> 2. People shouldn't be writing such complex comprehensions, so if the
>> rule doesn't give what they expect, they can always rewrite the code
>> with an explicit (and clearer) loop.
>>
>
> me too -- a direct translation to a for loop isn't necessary to understand
> how it works.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Spencer Brown]

The logical solution to me is to allow any order of while and if, and follow 
the same 'rule' as multiple for loops - just nest/test those in that order. 
Then you can have whatever priority you need. One question though is how this 
should handle multiple loops - break all of them, or just the current one?

- Spencer Brown

On 11 Aug 2017, at 6:27 am, Chris Barker 
> wrote:

On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore 
> wrote:

 Also, there's a potential issue
here - consider

[expr for var in even_numbers() if is_odd(var) while var < 100]

This is an infinite loop, even though it has a finite termination
condition (var < 100), because we only test the termination condition
if var is odd, which it never will be.

why is the termination only tested if teh if clause is True? Could then not be 
processed in parallel? or the while first

so maybe better to do:

[expr for var in even_numbers() while var < 100 if is_odd(var)]

Maybe it's just me, but I would certainly expect the while to have precedence.

I guess I think of it like this:

"if" is providing a filtering mechanism

"while" is providing a termination mechanism

 -- is there a use case anyone can think of when they would want the while to 
be applied to the list AFTER filtering?

Obviously, this is a contrived example. And certainly "don't do that,
then" is a valid response. But my instinct is that people are going to
get this wrong - *especially* in a maintenance environment.

sure, but would there be an issue if teh while were given precedence?

Overall, I agree with Steven's point. It seems pretty obvious what the
intention is, and while it's probably possible to construct examples
that are somewhat unclear,

1. The mechanical rule gives an explicit meaning
2. People shouldn't be writing such complex comprehensions, so if the
rule doesn't give what they expect, they can always rewrite the code
with an explicit (and clearer) loop.

me too -- a direct translation to a for loop isn't necessary to understand how 
it works.

-CHB


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Paul Moore]

On 10 August 2017 at 21:25, Chris Barker  wrote:
> On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:
>
>>
>>  Also, there's a potential issue
>> here - consider
>>
>> [expr for var in even_numbers() if is_odd(var) while var < 100]
>>
>> This is an infinite loop, even though it has a finite termination
>> condition (var < 100), because we only test the termination condition
>> if var is odd, which it never will be.
>
>
> why is the termination only tested if teh if clause is True? Could then not
> be processed in parallel? or the while first

See? That's my point - the "obvious" interpretation stops being
obvious pretty fast...

> so maybe better to do:
>
> [expr for var in even_numbers() while var < 100 if is_odd(var)]

That would work. But I bet people's intuition wouldn't immediately
lead to that fix (or indeed, necessarily incline them to put the
clauses in this order in the first place).

> Maybe it's just me, but I would certainly expect the while to have
> precedence.
>
> I guess I think of it like this:
>
> "if" is providing a filtering mechanism
>
> "while" is providing a termination mechanism
>
>  -- is there a use case anyone can think of when they would want the while
> to be applied to the list AFTER filtering?

Probably not, but when you can have multiple FORs, WHILEs and IFs, in
any order, explaining the behaviour precisely while still preserving
some sense of "filtering comes after termination" is going to be
pretty difficult. [expr for var1 in seq1 if cond1 for var2 in seq2 for
var3 in seq3 if cond2 if cond3] is legal - stupid, but legal. Now add
while clauses randomly in that, and define your expected semantics
clearly so a user (and the compiler!) can determine what the resulting
mess means.

The main benefit of the current "works like a for loop" interpretation
is that it's 100% explicit. Nothing will make a mess like the above
good code, but at least it's well-defined.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Thu, Aug 10, 2017 at 1:03 PM, Terry Reedy  wrote:

> After all, we have "break" in both for and while loops, so clearly there is
>>> the use case...
>>>
>>
> In both cases, we use 'break' to mean break.  If we want to break
> comprehensions, I think we should continue to use 'break' to mean break
> instead of twisting 'while' to mean 'break'.


I was thinking that too.


>> [expression for x in sequence if condition break]


hmm, but if you want to filter, also?

[expression for x in sequence if condition if condition break]

or

[expression for x in sequence if condition break if condition ]

both of those seem more confusing to me than while.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:


>  Also, there's a potential issue
> here - consider
>
> [expr for var in even_numbers() if is_odd(var) while var < 100]
>
> This is an infinite loop, even though it has a finite termination
> condition (var < 100), because we only test the termination condition
> if var is odd, which it never will be.
>

why is the termination only tested if teh if clause is True? Could then not
be processed in parallel? or the while first

so maybe better to do:

[expr for var in even_numbers() while var < 100 if is_odd(var)]

Maybe it's just me, but I would certainly expect the while to have
precedence.

I guess I think of it like this:

"if" is providing a filtering mechanism

"while" is providing a termination mechanism

 -- is there a use case anyone can think of when they would want the while
to be applied to the list AFTER filtering?

Obviously, this is a contrived example. And certainly "don't do that,
> then" is a valid response. But my instinct is that people are going to
> get this wrong - *especially* in a maintenance environment.


sure, but would there be an issue if teh while were given precedence?

Overall, I agree with Steven's point. It seems pretty obvious what the
> intention is, and while it's probably possible to construct examples
> that are somewhat unclear,
>
> 1. The mechanical rule gives an explicit meaning
> 2. People shouldn't be writing such complex comprehensions, so if the
> rule doesn't give what they expect, they can always rewrite the code
> with an explicit (and clearer) loop.
>

me too -- a direct translation to a for loop isn't necessary to understand
how it works.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Terry Reedy]

On 8/10/2017 9:42 AM, Steven D'Aprano wrote:

On Wed, Aug 09, 2017 at 01:23:28PM -0700, Chris Barker wrote:


I can't recall the use case(s) at the moment, but I have definitely wanted
a way to break out of a comprehension -- and not always with infinite
iterators.

After all, we have "break" in both for and while loops, so clearly there is
the use case...


In both cases, we use 'break' to mean break.  If we want to break 
comprehensions, I think we should continue to use 'break' to mean break 
instead of twisting 'while' to mean 'break'.



[expression for x in sequence while condition]

should (I believe) be obvious to anyone who already groks comprehension
syntax. The mapping to a for-loop is admittedly a tad more complex:

result = []
for x in sequence:
 if not condition: break
 result.append(expression)


This is the same as

result = []
for x in sequence:
if condition:
result.append(expression)
else:
break

which could be written

[expression for x in sequence if condition break]

--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Soni L.]

On 2017-08-10 01:11 PM, Steven D'Aprano wrote:

On Thu, Aug 10, 2017 at 12:39:34PM -0300, Soni L. wrote:


I'm pretty sure I read somewhere that lambdas and generators share their
syntax, and that syntax is already a subset of python syntax. Would it
be too hard to expose that with a "simplified AST" API?

I don't understand what you mean by this.

The syntax for lambda is (roughly):

  lambda parameter-list : expression

The syntax for generators is (again, roughly):

  def name ( parameter-list ) :
  suite-containing-yield


Obviously the generator suite can contain expressions, and both have a
parameter-list. What shared syntax are you referring to, and how is it
relevant?

Or are you referring to generator expressions, rather than generators?

  ( expression for target in expression ... )

Obviously a Python expression is a Python expression, wherever it is, so
a lambda can contain generator expressions, and generator expressions
can contain lambdas...

And what do you mean by "simplified AST" API? I'm afraid your comment is
too abstract for me to understand.



Yes, see, both are expressions. Expression AST is a subset of python 
AST, so it's a simplified form of the python AST.






___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Soni L.]

On 2017-08-10 01:43 AM, Nick Coghlan wrote:

On 10 August 2017 at 01:49, Soni L.  wrote:

On 2017-08-09 11:54 AM, Nick Coghlan wrote:

Right, I was separating the original request to make "{x for x in
integers if 1000 <= x < 100}" work into the concrete proposal to
make exactly *that* syntax work (which I don't think is feasible), and
the slightly more general notion of offering a more math-like syntax
that allows finite sets to be built from infinite iterators by
defining a termination condition in addition to a filter condition.

Ok. A concrete proposal would give a read-only 'filter' argument to the
iterator somehow, which represents some form of simplified AST of the
condition.

So e.g. {x for x in integers if (lambda v: 1000 <= v < 100)(x)} would
never complete, but {x for x in integers if 1000 <= x < 100} would. (But
perhaps lambda objects should include an AST attribute... Having it for
normal functions would introduce too much overhead tho, and then it would no
longer be a simplified AST, but rather a complete python AST, which we don't
want.)

There have been a variety of different "thunking" proposals over the
years, but they've all foundered on the question of what the
*primitive* quoted form should look like, and how the thunks should
subsequently be executed.

For cases like this, where integration with Python's name resolution
mechanism isn't actually required, folks have ended up just using
strings, where the only downside is the fact that syntax highlighters
and other static analysers don't know that the contents are supposed
to be valid Python code. In a case like this, that might look like:

 {x for x in integers.build_set("1000 <= x < 100")}

As with regexes, the cost of dynamically parsing such strings can then
be amortised at runtime through the use of an appropriate caching
strategy.


I'm pretty sure I read somewhere that lambdas and generators share their 
syntax, and that syntax is already a subset of python syntax. Would it 
be too hard to expose that with a "simplified AST" API?




Cheers,
Nick.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Paul Moore]

On 10 August 2017 at 14:42, Steven D'Aprano  wrote:
> I don't think it is confusing. Regardless of the implementation, the
> meaning of:
>
> [expression for x in sequence while condition]
>
> should (I believe) be obvious to anyone who already groks comprehension
> syntax. The mapping to a for-loop is admittedly a tad more complex:
>
> result = []
> for x in sequence:
> if not condition: break
> result.append(expression)
>
> but I'm yet to meet anyone who routinely and regularly reads
> comprehensions by converting them to for loops like that. And if they
> did, all they need do is mentally map "while condition" to "if not
> condition: break" and it should all Just Work™.

The hard part is the interaction between if and while.

Consider (expr for var in seq if cond1 while cond2):

This means:

for var in seq:
if cond1:
if not cond2: break
yield expr

Note that unlike all other comprehension clauses (for and if) while
doesn't introduce a new level of nesting. That's an inconsistency, and
while it's minor, it would need clarifying (my original draft of this
email was a mess, because I misinterpreted how if and while would
interact, precisely over this point). Also, there's a potential issue
here - consider

[expr for var in even_numbers() if is_odd(var) while var < 100]

This is an infinite loop, even though it has a finite termination
condition (var < 100), because we only test the termination condition
if var is odd, which it never will be.

Obviously, this is a contrived example. And certainly "don't do that,
then" is a valid response. But my instinct is that people are going to
get this wrong - *especially* in a maintenance environment. That
example could have started off being "for var in count(0)" and then
someone realised they could "optimise" it by omitting odd numbers,
introducing the bug in the process. (And I'm sure real life code could
come up with much subtler examples ;-))

Overall, I agree with Steven's point. It seems pretty obvious what the
intention is, and while it's probably possible to construct examples
that are somewhat unclear,

1. The mechanical rule gives an explicit meaning
2. People shouldn't be writing such complex comprehensions, so if the
rule doesn't give what they expect, they can always rewrite the code
with an explicit (and clearer) loop.

But while I think this says that the above interpretation of while is
the only sensible one, and in general other approaches are unlikely to
be as natural, I *don't* think that it unequivocally says that
allowing while is a good thing. It may still be better to omit it, and
force people to state their intent explicitly (albeit a bit more
verbosely).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Thu, Aug 10, 2017 at 12:54:57AM +1000, Nick Coghlan wrote:

Guido wrote:
> > I haven't come around to this yet. It looks like it will make explaining
> > comprehensions more complex, since the translation of "while X" into "if not
> > X: break" feels less direct than the translations of "for x in xs" or "if
> > pred(x)". (In particular, your proposal seems to require more experience
> > with mentally translating loops and conditions into jumps -- most regulars
> > of this forum do that for a living, but I doubt it's second nature for the
> > OP.)
>
> Yeah, if we ever did add something like this, I suspect a translation
> using takewhile would potentially be easier for at least some users to
> understand than the one to a break condition:

"Some users"? Sure, why not? There's probably somebody out there who 
understands takewhile, but if so, I don't know who they are :-)

I always have to look at the docs for takewhile to remind myself whether 
it drops items ("takes them away") while the condition is true, or 
yields items ("gives items") while the condition is true.


> {x for x in itertools.count(0) if 1000 <= x while x < 100}
> 
> <=>
> 
> x = set()
> for x in itertools.count(0):
> if 1000 <= x:
> set.add(x)
> # If you've never used the loop-and-a-half idiom, it's
> # not obvious why "while " means "if not : break"
> if not x < 100:
> break

I'd like to take issue with that "not obvious" comment. I think that 
anyone who knows while loops knows that the loop exits when the 
condition becomes false. That's exactly the behaviour we get for the 
(hypothetical) [expr for x in seq while condition] syntax: when the 
condition is false, the loop and hence the comprehension, exits.

For such simple cases, there's no need to think about "loop and a half". 
The obvious explanation is that the loop exits when the while condition 
fails.

Based on my experience with beginners on the tutor mailing list, and 
elsewhere, I think there's a definite learning "hump" to get over before 
people grok even the trivial case of

[expression for x in sequence]

but once they do, then adding an "if" clause is obvious, and I expect 
that the same will apply to "when".

Once you move beyond the simple case of a single for and no more than a 
single if (or while), I don't think there's *anything* obvious about 
comprehension syntax at all, while clause or no while clause. Holding 
the while clause to a standard that comprehensions already fail (in my 
opinion) is unfair:

[expression for x in seq1 for y in seq2 if pred1 for z in seq3 
if pred2 if pred3 if pred4 for w in seq4 while condition for v in seq5]

I don't think it's the "while" that tips that over the edge, 
readability-wise :-)


In any case, I think we're all guessing whether or not people will 
understand the "while condition" syntax. So I've done an informal survey 
on the Python-Ideas list, and once folks have had a day or so to answer 
I'll report what they say. It's not a truly scientific UI test, but it's 
the best I can do.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Wed, Aug 09, 2017 at 01:23:28PM -0700, Chris Barker wrote:

> I can't recall the use case(s) at the moment, but I have definitely wanted
> a way to break out of a comprehension -- and not always with infinite
> iterators.
>
> After all, we have "break" in both for and while loops, so clearly there is
> the use case...

Indeed :-)


> If someone comes up with a clean and not confusing (and general purpose)
> syntax, I think it would be very useful.

We used to be able to (ab)use StopIteration to do this:

def Break():
raise StopIteration

# generator expressions only, not list comprehensions
result = (expression for x in sequence if condition or Break())

but I believe that loophole has been closed in 3.6.


Comprehensions in Clojure have this feature:

http://clojuredocs.org/clojure_core/clojure.core/for

Clojure uses "when" where Python uses "if", giving:

;; :when continues through the collection even if some have the
;; condition evaluate to false, like filter
user=> (for [x (range 3 33 2) :when (prime? x)]
 x)
(3 5 7 11 13 17 19 23 29 31)

;; :while stops at the first collection element that evaluates to
;; false, like take-while
user=> (for [x (range 3 33 2) :while (prime? x)]
 x)
(3 5 7)


Translating into Python:

[x for x in range(3, 33, 2) if is_prime(x)]

[x for x in range(3, 33, 2) while is_prime(x)]  # hypothetical syntax

I don't think it is confusing. Regardless of the implementation, the 
meaning of:

[expression for x in sequence while condition]

should (I believe) be obvious to anyone who already groks comprehension 
syntax. The mapping to a for-loop is admittedly a tad more complex:

result = []
for x in sequence:
if not condition: break
result.append(expression)

but I'm yet to meet anyone who routinely and regularly reads 
comprehensions by converting them to for loops like that. And if they 
did, all they need do is mentally map "while condition" to "if not 
condition: break" and it should all Just Work™.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 10 August 2017 at 01:49, Soni L.  wrote:
> On 2017-08-09 11:54 AM, Nick Coghlan wrote:
>> Right, I was separating the original request to make "{x for x in
>> integers if 1000 <= x < 100}" work into the concrete proposal to
>> make exactly *that* syntax work (which I don't think is feasible), and
>> the slightly more general notion of offering a more math-like syntax
>> that allows finite sets to be built from infinite iterators by
>> defining a termination condition in addition to a filter condition.
>
> Ok. A concrete proposal would give a read-only 'filter' argument to the
> iterator somehow, which represents some form of simplified AST of the
> condition.
>
> So e.g. {x for x in integers if (lambda v: 1000 <= v < 100)(x)} would
> never complete, but {x for x in integers if 1000 <= x < 100} would. (But
> perhaps lambda objects should include an AST attribute... Having it for
> normal functions would introduce too much overhead tho, and then it would no
> longer be a simplified AST, but rather a complete python AST, which we don't
> want.)

There have been a variety of different "thunking" proposals over the
years, but they've all foundered on the question of what the
*primitive* quoted form should look like, and how the thunks should
subsequently be executed.

For cases like this, where integration with Python's name resolution
mechanism isn't actually required, folks have ended up just using
strings, where the only downside is the fact that syntax highlighters
and other static analysers don't know that the contents are supposed
to be valid Python code. In a case like this, that might look like:

{x for x in integers.build_set("1000 <= x < 100")}

As with regexes, the cost of dynamically parsing such strings can then
be amortised at runtime through the use of an appropriate caching
strategy.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 10 August 2017 at 00:54, Nick Coghlan  wrote:
> Yeah, if we ever did add something like this, I suspect a translation
> using takewhile would potentially be easier for at least some users to
> understand than the one to a break condition:
>
> {x for x in itertools.count(0) if 1000 <= x while x < 100}
>
> <=>
>
> x = set()
> for x in itertools.count(0):
> if 1000 <= x:
> set.add(x)
> # If you've never used the loop-and-a-half idiom, it's
> # not obvious why "while " means "if not : break"
> if not x < 100:
> break
>
> is roughly
>
> {x for x in itertools.takewhile(itertools.count(0), lambda x: x <
> 100) if 1000 <= x}
>
> <=>
>
> x = set()
> for x in takewhile(itertools.count(0), lambda x: x < 100):
> if 1000 <= x:
> set.add(x)

Ugh, this discrepancy is worse than I thought, since the translation
with that clause order is actually wrong (Terry mentioned this by
pointing out that the proposed syntactic translation implemented
"do...while" ordering). The takewhile example is also wrong, since it
has the arguments in the wrong order. Fixing both of those issues
gives the comparison:

{x for x in itertools.count(0) while x < 100 if 1000 <= x}

<=>

x = set()
for x in itertools.count(0):
# If you've never used the loop-and-a-half idiom, it's
# not obvious why "while " means "if : 
else: break"
if x < 100:
if 1000 <= x:
set.add(x)
else:
break

is roughly:

{x for x in itertools.takewhile(lambda x: x < 100,
itertools.count(0)) if 1000 <= x}

<=>

x = set()
for x in takewhile(lambda x: x < 100, itertools.count(0)):
if 1000 <= x:
set.add(x)

And I think that gets me back to pretty much where I was the last time
this came up: a while clause in comprehensions really only makes sense
in combination with a while clause on for loops, where:

for x in itertools.count(0) while x < 100:
...

was roughly equivalent to:

for x in itertools.count(0):
if x < 100:
...
else:

break

(such that there's only one loop from the point of view of
break/continue/else, but the loop may terminate based on either
exhaustion of the underlying iterator *or* some specific condition
becoming false)

While I do think such a clause would be more readable for more people
than the dropwhile/takewhile equivalents (especially when the latter
end up needing to use lambda expressions), I'm still dubious that
these cases come up often enough to justify the addition of a
for-while loop as a composite construct (the old "dropwhile and
takewhile aren't even common enough to justify being builtins, why
should they jump all the way to syntactic support?" question applies).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Terry Reedy]

On 8/9/2017 10:54 AM, Nick Coghlan wrote:

On 9 August 2017 at 15:38, Guido van Rossum  wrote:

On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:

The OP's proposal doesn't fit into that category though: rather it's
asking about the case where we have an infinite iterator (e.g.
itertools.count(0)), and want to drop items until they start meeting
some condition (i.e. itertools.dropwhile) and then terminate the
iterator as soon as another condition is no longer met (i.e.
itertools.takewhile).


I don't think that's what the OP meant. The original proposal seemed to
assume that it would be somehow reasonable for the input ("integers" in the
example) to be able to see and parse the condition in the generator
expression ("1000 <= x < 10" in the example, with "x" somehow known to
be bound to the iteration value). That's at least what I think the remark "I
like mathy syntax" referred to.


Right, I was separating the original request to make "{x for x in
integers if 1000 <= x < 100}" work into the concrete proposal to
make exactly *that* syntax work (which I don't think is feasible), and
the slightly more general notion of offering a more math-like syntax
that allows finite sets to be built from infinite iterators by
defining a termination condition in addition to a filter condition.


We already have three nice one liners for that, one of which you gave.

x = set(filter(filter_condition, takewhile(continue_condition, source)))
x = set(x for x in takewhile(continue_condition, source) if 
filter_condition)

x = {x for x in takewhile(continue_condition, source) if filter_condition}

Replace takewhile with islice(source, max) if the continue condition is 
(number seen < max).  Add enumerate if the running count is needed 
otherwise.


Terminating an infinite iterator and filtering the initial slice are 
different operations.  The operations are easily composed as they are, 
in multiple ways.  Trying to mix them together in one jumbled special 
syntax is a bad idea to me.



There aren't any technical barriers I'm aware of to implementing that,
with the main historical objection being that instead of the
comprehension level while clause mapping to a while loop directly the
way the for and if clauses map to their statement level counterparts,
it would instead map to the conditional break in the expanded
loop-and-a-half form:

 while True:

 

 if not condition:
 break


In other words, aside from other issues, you would have 'while' mean 
'do...while' in this one special place.  -1.


--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:

> On 8 August 2017 at 09:06, Chris Barker  wrote:
> > It would be nice to have an easier access to an "slice iterator" though
> --
> > one of these days I may write up a proposal for that.
>
> An idea I've occasionally toyed with [1] is some kind of "iterview"
> that wraps around an arbitrary iterable and produces lazy itertools
> based results rather than immediate views or copies.
>
> However, my experience is also that folks are *really* accustomed to
> syntactic operations on containers producing either full live views
> (e.g. memoryview or numpy slices, range as a dynamically computed
> container), or actual copies (builtin container types). Having them
> produce consumable iterators instead then gets confusing due to the
> number of operations that will implicitly consume them (including
> simple "x in y" checks).
>

I agree -- which is why I"m thinking only adding a simple "iterable slice",
rather than changing the overall behavior of the container. It would be
quite clear what you are asking for.

Right now, getting the "terminate when false" behaviour requires the
> use of takewhile:
>

I can't recall the use case(s) at the moment, but I have definitely wanted
a way to break out of a comprehension -- and not always with infinite
iterators.

After all, we have "break" in both for and while loops, so clearly there is
the use case...

If someone comes up with a clean and not confusing (and general purpose)
syntax, I think it would be very useful.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Soni L.]

On 2017-08-09 11:54 AM, Nick Coghlan wrote:

On 9 August 2017 at 15:38, Guido van Rossum  wrote:

On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:

The OP's proposal doesn't fit into that category though: rather it's
asking about the case where we have an infinite iterator (e.g.
itertools.count(0)), and want to drop items until they start meeting
some condition (i.e. itertools.dropwhile) and then terminate the
iterator as soon as another condition is no longer met (i.e.
itertools.takewhile).

I don't think that's what the OP meant. The original proposal seemed to
assume that it would be somehow reasonable for the input ("integers" in the
example) to be able to see and parse the condition in the generator
expression ("1000 <= x < 10" in the example, with "x" somehow known to
be bound to the iteration value). That's at least what I think the remark "I
like mathy syntax" referred to.

Right, I was separating the original request to make "{x for x in
integers if 1000 <= x < 100}" work into the concrete proposal to
make exactly *that* syntax work (which I don't think is feasible), and
the slightly more general notion of offering a more math-like syntax
that allows finite sets to be built from infinite iterators by
defining a termination condition in addition to a filter condition.


Ok. A concrete proposal would give a read-only 'filter' argument to the 
iterator somehow, which represents some form of simplified AST of the 
condition.


So e.g. {x for x in integers if (lambda v: 1000 <= v < 100)(x)} 
would never complete, but {x for x in integers if 1000 <= x < 100} 
would. (But perhaps lambda objects should include an AST attribute... 
Having it for normal functions would introduce too much overhead tho, 
and then it would no longer be a simplified AST, but rather a complete 
python AST, which we don't want.)





There aren't any technical barriers I'm aware of to implementing that,
with the main historical objection being that instead of the
comprehension level while clause mapping to a while loop directly the
way the for and if clauses map to their statement level counterparts,
it would instead map to the conditional break in the expanded
loop-and-a-half form:

 while True:
 if not condition:
 break

While it's taken me a long time to come around to the idea, "Make
subtle infinite loops in mathematical code easier to avoid" *is* a
pretty compelling user-focused justification for incurring that extra
complexity at the language design level.

I haven't come around to this yet. It looks like it will make explaining
comprehensions more complex, since the translation of "while X" into "if not
X: break" feels less direct than the translations of "for x in xs" or "if
pred(x)". (In particular, your proposal seems to require more experience
with mentally translating loops and conditions into jumps -- most regulars
of this forum do that for a living, but I doubt it's second nature for the
OP.)

Yeah, if we ever did add something like this, I suspect a translation
using takewhile would potentially be easier for at least some users to
understand than the one to a break condition:

 {x for x in itertools.count(0) if 1000 <= x while x < 100}

 <=>

 x = set()
 for x in itertools.count(0):
 if 1000 <= x:
 set.add(x)
 # If you've never used the loop-and-a-half idiom, it's
 # not obvious why "while " means "if not : break"
 if not x < 100:
 break

 is roughly

 {x for x in itertools.takewhile(itertools.count(0), lambda x: x <
100) if 1000 <= x}

 <=>

 x = set()
 for x in takewhile(itertools.count(0), lambda x: x < 100):
 if 1000 <= x:
 set.add(x)

However, the break condition is the translation that would make sense
at a language *implementation* level (and would hence be the one that
determined the relative location of the while clause in the expression
form).

That discrepancy *still* sets off alarm bells for me (since it's a
clear sign that "how people would think this works" and "how it would
actually work" probably wouldn't match), I'm also conscious of the
amount of syntactic noise that "takewhile" introduces vs the "while"
keyword.

The counter-argument (which remains valid even against my own change
of heart) is that adding a new comprehension clause doesn't actually
fix the "accidental infinite loop" problem: "{x for x in
itertools.count(0) if 1000 <= x < 100}" will still loop forever,
it would just have a nicer fix to get it to terminate (adding " while
x" to turn the second filter condition into a termination condition).

So while I'm +0 where I used to be a firm -1, it's still only a +0 :)

Cheers,
Nick.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: 

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 9 August 2017 at 15:38, Guido van Rossum  wrote:
> On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:
>> The OP's proposal doesn't fit into that category though: rather it's
>> asking about the case where we have an infinite iterator (e.g.
>> itertools.count(0)), and want to drop items until they start meeting
>> some condition (i.e. itertools.dropwhile) and then terminate the
>> iterator as soon as another condition is no longer met (i.e.
>> itertools.takewhile).
>
> I don't think that's what the OP meant. The original proposal seemed to
> assume that it would be somehow reasonable for the input ("integers" in the
> example) to be able to see and parse the condition in the generator
> expression ("1000 <= x < 10" in the example, with "x" somehow known to
> be bound to the iteration value). That's at least what I think the remark "I
> like mathy syntax" referred to.

Right, I was separating the original request to make "{x for x in
integers if 1000 <= x < 100}" work into the concrete proposal to
make exactly *that* syntax work (which I don't think is feasible), and
the slightly more general notion of offering a more math-like syntax
that allows finite sets to be built from infinite iterators by
defining a termination condition in addition to a filter condition.

>> There aren't any technical barriers I'm aware of to implementing that,
>> with the main historical objection being that instead of the
>> comprehension level while clause mapping to a while loop directly the
>> way the for and if clauses map to their statement level counterparts,
>> it would instead map to the conditional break in the expanded
>> loop-and-a-half form:
>>
>> while True:
>> if not condition:
>> break
>>
>> While it's taken me a long time to come around to the idea, "Make
>> subtle infinite loops in mathematical code easier to avoid" *is* a
>> pretty compelling user-focused justification for incurring that extra
>> complexity at the language design level.
>
> I haven't come around to this yet. It looks like it will make explaining
> comprehensions more complex, since the translation of "while X" into "if not
> X: break" feels less direct than the translations of "for x in xs" or "if
> pred(x)". (In particular, your proposal seems to require more experience
> with mentally translating loops and conditions into jumps -- most regulars
> of this forum do that for a living, but I doubt it's second nature for the
> OP.)

Yeah, if we ever did add something like this, I suspect a translation
using takewhile would potentially be easier for at least some users to
understand than the one to a break condition:

{x for x in itertools.count(0) if 1000 <= x while x < 100}

<=>

x = set()
for x in itertools.count(0):
if 1000 <= x:
set.add(x)
# If you've never used the loop-and-a-half idiom, it's
# not obvious why "while " means "if not : break"
if not x < 100:
break

is roughly

{x for x in itertools.takewhile(itertools.count(0), lambda x: x <
100) if 1000 <= x}

<=>

x = set()
for x in takewhile(itertools.count(0), lambda x: x < 100):
if 1000 <= x:
set.add(x)

However, the break condition is the translation that would make sense
at a language *implementation* level (and would hence be the one that
determined the relative location of the while clause in the expression
form).

That discrepancy *still* sets off alarm bells for me (since it's a
clear sign that "how people would think this works" and "how it would
actually work" probably wouldn't match), I'm also conscious of the
amount of syntactic noise that "takewhile" introduces vs the "while"
keyword.

The counter-argument (which remains valid even against my own change
of heart) is that adding a new comprehension clause doesn't actually
fix the "accidental infinite loop" problem: "{x for x in
itertools.count(0) if 1000 <= x < 100}" will still loop forever,
it would just have a nicer fix to get it to terminate (adding " while
x" to turn the second filter condition into a termination condition).

So while I'm +0 where I used to be a firm -1, it's still only a +0 :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Stephen J. Turnbull]

Nick Coghlan writes:

 > Right now, getting the "terminate when false" behaviour requires the
 > use of takewhile:
 > 
 > {itertools.takewhile(lambda x: x < 100, itertools.count(1000)}

My objection to this interpretation is different from Guido's (I
think): if you're really thinking in terms of math, sets are
*unordered*, and therefore "takewhile" doesn't guarantee exhaustion of
the desired subset.  Another way to put this is that in order to make
it harder to get bit by subtle infloops, you're going to give more
teeth to "Miller time came early"[1] bugs.

This may be a bigger issue than some may think, because sets and dicts
are iterable, and order of iteration is arbitrary (at best history-
dependent).


Footnotes: 
[1]  American beer commercial claiming that real men go to drink beer
after a full day's work.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Guido van Rossum]

On Tue, Aug 8, 2017 at 10:06 PM, Nick Coghlan  wrote:

> On 8 August 2017 at 09:06, Chris Barker  wrote:
> > It would be nice to have an easier access to an "slice iterator" though
> --
> > one of these days I may write up a proposal for that.
>
> An idea I've occasionally toyed with [1] is some kind of "iterview"
> that wraps around an arbitrary iterable and produces lazy itertools
> based results rather than immediate views or copies.
>
> However, my experience is also that folks are *really* accustomed to
> syntactic operations on containers producing either full live views
> (e.g. memoryview or numpy slices, range as a dynamically computed
> container), or actual copies (builtin container types). Having them
> produce consumable iterators instead then gets confusing due to the
> number of operations that will implicitly consume them (including
> simple "x in y" checks).
>
> The OP's proposal doesn't fit into that category though: rather it's
> asking about the case where we have an infinite iterator (e.g.
> itertools.count(0)), and want to drop items until they start meeting
> some condition (i.e. itertools.dropwhile) and then terminate the
> iterator as soon as another condition is no longer met (i.e.
> itertools.takewhile).
>

I don't think that's what the OP meant. The original proposal seemed to
assume that it would be somehow reasonable for the input ("integers" in the
example) to be able to see and parse the condition in the generator
expression ("1000 <= x < 10" in the example, with "x" somehow known to
be bound to the iteration value). That's at least what I think the remark
"I like mathy syntax" referred to.


> Right now, getting the "terminate when false" behaviour requires the
> use of takewhile:
>
> {itertools.takewhile(lambda x: x < 100, itertools.count(1000)}
>
> In these cases, the standard generator expression syntax is an
> attractive nuisance because it *looks* right from a mathematical
> perspective, but hides an infinite loop:
>
> {x for x in itertools.count(0) if 1000 <= x < 100}
>
> The most credible proposal to address this that I've seen is to borrow
> the "while" keyword in its "if not x: break" interpretation to get:
>
> {x for x in itertools.count(0) if 1000 <= x while x < 100}
>
> which would be compiled as equivalent to:
>
> x = set()
> for x in itertools.count(0):
> if 1000 <= x:
> set.add(x)
> if not x < 100:
> break
>
> (and similarly for all of the other comprehension variants)
>
> There aren't any technical barriers I'm aware of to implementing that,
> with the main historical objection being that instead of the
> comprehension level while clause mapping to a while loop directly the
> way the for and if clauses map to their statement level counterparts,
> it would instead map to the conditional break in the expanded
> loop-and-a-half form:
>
> while True:
> if not condition:
> break
>
> While it's taken me a long time to come around to the idea, "Make
> subtle infinite loops in mathematical code easier to avoid" *is* a
> pretty compelling user-focused justification for incurring that extra
> complexity at the language design level.
>

I haven't come around to this yet. It looks like it will make explaining
comprehensions more complex, since the translation of "while X" into "if
not X: break" feels less direct than the translations of "for x in xs" or
"if pred(x)". (In particular, your proposal seems to require more
experience with mentally translating loops and conditions into jumps --
most regulars of this forum do that for a living, but I doubt it's second
nature for the OP.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 8 August 2017 at 09:06, Chris Barker  wrote:
> It would be nice to have an easier access to an "slice iterator" though --
> one of these days I may write up a proposal for that.

An idea I've occasionally toyed with [1] is some kind of "iterview"
that wraps around an arbitrary iterable and produces lazy itertools
based results rather than immediate views or copies.

However, my experience is also that folks are *really* accustomed to
syntactic operations on containers producing either full live views
(e.g. memoryview or numpy slices, range as a dynamically computed
container), or actual copies (builtin container types). Having them
produce consumable iterators instead then gets confusing due to the
number of operations that will implicitly consume them (including
simple "x in y" checks).

The OP's proposal doesn't fit into that category though: rather it's
asking about the case where we have an infinite iterator (e.g.
itertools.count(0)), and want to drop items until they start meeting
some condition (i.e. itertools.dropwhile) and then terminate the
iterator as soon as another condition is no longer met (i.e.
itertools.takewhile).

Right now, getting the "terminate when false" behaviour requires the
use of takewhile:

{itertools.takewhile(lambda x: x < 100, itertools.count(1000)}

In these cases, the standard generator expression syntax is an
attractive nuisance because it *looks* right from a mathematical
perspective, but hides an infinite loop:

{x for x in itertools.count(0) if 1000 <= x < 100}

The most credible proposal to address this that I've seen is to borrow
the "while" keyword in its "if not x: break" interpretation to get:

{x for x in itertools.count(0) if 1000 <= x while x < 100}

which would be compiled as equivalent to:

x = set()
for x in itertools.count(0):
if 1000 <= x:
set.add(x)
if not x < 100:
break

(and similarly for all of the other comprehension variants)

There aren't any technical barriers I'm aware of to implementing that,
with the main historical objection being that instead of the
comprehension level while clause mapping to a while loop directly the
way the for and if clauses map to their statement level counterparts,
it would instead map to the conditional break in the expanded
loop-and-a-half form:

while True:
if not condition:
break

While it's taken me a long time to come around to the idea, "Make
subtle infinite loops in mathematical code easier to avoid" *is* a
pretty compelling user-focused justification for incurring that extra
complexity at the language design level.

Cheers,
Nick.

[1] https://mail.python.org/pipermail/python-ideas/2010-April/006983.html

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Stephen J. Turnbull]

> Soni L. writes:
 > Steven d'Aprano writes:

 > > range(1000, 100)
 > > (x for x in range(1000, 100))  # waste of time and effort

 > Actually, those have different semantics!

That's not real important.  As Stefan Behnel points out, it's simple
(and efficient) to get iterator semantics by using iter().

The big issue here is that Python is not the kind of declarative
language where (x for x in int if 1_000 ≤ x ≤ 1_000_000)[1] is
natural to write, let alone easy to implement efficiently.  Aside from
the problem of (x for x in float if 1_000 ≤ x ≤ 1_000_000) (where
the answer is "just don't do that"), I can't think of any unbounded
collections in Python that aren't iterables, except some types.

That makes Steven's criticism pretty compelling.  If you need to
design a collection's __iter__ specially to allow it to decide whether
the subset that satisfies some condition is exhausted, why not just
subclass some appropriate existing collection with a more appropriate
__iter__?

Footnotes: 
[1]  See what I did there? ;-)

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Stefan Behnel]

Soni L. schrieb am 08.08.2017 um 01:56:
> On 2017-08-07 08:35 PM, Steven D'Aprano wrote:
>> Hi Soni, and welcome!
>>
>> On Mon, Aug 07, 2017 at 04:30:05PM -0300, Soni L. wrote:
>>
>>> What if, (x for x in integers if 1000 <= x < 100), was syntax sugar
>>> for (x for x in range(1000, 100))?
>> If you want the integers from 1000 to 100, use:
>>
>> range(1000, 100)
>>
>> Don't waste your time slowing down the code with an unnecessary and
>> pointless wrapper that does nothing but pass every value on unchanged:
>>
>> (x for x in range(1000, 100))  # waste of time and effort
> 
> Actually, those have different semantics!
> 
 x = range(1, 10)
 list(x)
> [1, 2, 3, 4, 5, 6, 7, 8, 9]
 list(x)
> [1, 2, 3, 4, 5, 6, 7, 8, 9]
> 
 x = (x for x in range(1, 10))
 list(x)
> [1, 2, 3, 4, 5, 6, 7, 8, 9]
 list(x)
> []

In that case, use iter(range(1000, 100)).

range() creates an iterable, which is iterable more than once.

iter(range()) creates an iterator from that iterable, which has the
semantics that you apparently wanted.

Stefan

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Soni L.]

On 2017-08-07 08:35 PM, Steven D'Aprano wrote:

Hi Soni, and welcome!

On Mon, Aug 07, 2017 at 04:30:05PM -0300, Soni L. wrote:


What if, (x for x in integers if 1000 <= x < 100), was syntax sugar
for (x for x in range(1000, 100))?

If you want the integers from 1000 to 100, use:

range(1000, 100)

Don't waste your time slowing down the code with an unnecessary and
pointless wrapper that does nothing but pass every value on unchanged:

(x for x in range(1000, 100))  # waste of time and effort





Actually, those have different semantics!

>>> x = range(1, 10)
>>> list(x)
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(x)
[1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> x = (x for x in range(1, 10))
>>> list(x)
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(x)
[]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

Hi Soni, and welcome!

On Mon, Aug 07, 2017 at 04:30:05PM -0300, Soni L. wrote:

> What if, (x for x in integers if 1000 <= x < 100), was syntax sugar 
> for (x for x in range(1000, 100))?

If you want the integers from 1000 to 100, use:

range(1000, 100)

Don't waste your time slowing down the code with an unnecessary and 
pointless wrapper that does nothing but pass every value on unchanged:

(x for x in range(1000, 100))  # waste of time and effort



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Mon, Aug 7, 2017 at 4:14 PM, Chris Angelico  wrote:

> On Tue, Aug 8, 2017 at 5:30 AM, Soni L.  wrote:
> > The generator syntax, (x for x in i if c), currently always creates a new
> > generator.


that's what it's for -- I'm confused as to what the problem is.


> > {x for x in integers if 1000 <= x < 100} # never completes, because
> it's
> > trying to iterate over all integers
>

this is a set comprehension -- but what is "integers"? is it a generator?

in which case,  it should take an argument so it knows when to end. Or if
it's really that symple, that's what range() is for.


However, similarly, I find that sometimes I want to iterate over a slice of
a sequence, but do'nt want to actually make the slice first.

So there is itertools.islice()

If "integers" is a sequence:

{x for x in integers[1000:1]}

makes an unneeded copy of that slice.

{x for x in itertools.islice(integers, 1000, 1)}

will iterate on the fly, and not make any extra copies.

It would be nice to have an easier access to an "slice iterator" though --
one of these days I may write up a proposal for that.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Angelico]

On Tue, Aug 8, 2017 at 5:30 AM, Soni L.  wrote:
> The generator syntax, (x for x in i if c), currently always creates a new
> generator. I find this quite inefficient:
>
> {x for x in integers if 1000 <= x < 100} # never completes, because it's
> trying to iterate over all integers
>
> What if, somehow, object `integers` could hook the generator and produce the
> equivalent of {x for x in range(1000, 100)}, which does complete?
>
> What if, (x for x in integers if 1000 <= x < 100), was syntax sugar for
> (x for x in range(1000, 100))?
>
> (I like mathy syntax. Do you like mathy syntax?)

I don't. I prefer to stick with the syntax we already have. The
alternative is a more verbose way to identify a range, plus you need a
new global "integers" which implies that you could iterate over
"reals" the same way (after all, mathematics doesn't mind you working
with a subset of reals the same way you'd work with a subset of ints).
And good luck iterating over all the reals. :)

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Generator syntax hooks?

[Soni L.]

The generator syntax, (x for x in i if c), currently always creates a 
new generator. I find this quite inefficient:


{x for x in integers if 1000 <= x < 100} # never completes, because 
it's trying to iterate over all integers


What if, somehow, object `integers` could hook the generator and produce 
the equivalent of {x for x in range(1000, 100)}, which does complete?


What if, (x for x in integers if 1000 <= x < 100), was syntax sugar 
for (x for x in range(1000, 100))?


(I like mathy syntax. Do you like mathy syntax?)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/