Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 11 August 2017 at 15:13, Steven D'Aprano  wrote:
> On Fri, Aug 11, 2017 at 02:34:53PM +1000, Nick Coghlan wrote:
>> This is another good reason why a termination condition would need to
>> be checked before the filter condition rather than either after it, or
>> only when the filter condition was true.
>
> Why is this a problem that needs solving?

Because the most obvious interpretation of a completely independent
"while" clause in comprehensions would be as a nested loop inside the
outer for loop, not as a nested if-else-break statement.

As a result of that, I'm only personally prepared to support for-while
comprehensions if they're syntactic sugar for a combined statement
level for-while loop that makes it clear why only the "for" clauses in
a comprehension create new loops.

I *wouldn't* be prepared to support them if they could only be
explained in terms of a direct mapping to an if statement and had no
statement level counterpart that actually used the "while" keyword.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Fri, Aug 11, 2017 at 02:49:10PM +1000, Steven D'Aprano wrote:
> On Thu, Aug 10, 2017 at 01:25:24PM -0700, Chris Barker wrote:

> > I guess I think of it like this:
> > 
> > "if" is providing a filtering mechanism
> > 
> > "while" is providing a termination mechanism
> > 
> >  -- is there a use case anyone can think of when they would want the while
> > to be applied to the list AFTER filtering?


Oops, sorry I had a thinko and read your question in the opposite sense 
than it actually is. See my response to Nick for an example:

I have an iterable of arbitrary objects. I want to ignore anything that
isn't a string, and halt if the string doesn't start with "A".

[expr for s in objects if isinstance(s, str) while s.startswith("A")]


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Fri, Aug 11, 2017 at 02:34:53PM +1000, Nick Coghlan wrote:

> This is actually how I came to the conclusion that if we were ever to
> do something like this, the termination condition would need to go
> *before* the filter condition:

What if you want to check the filter condition before the termination 
condition?

I have an iterable of arbitrary objects. I want to ignore anything that 
isn't a string, and halt if the string doesn't start with "A". This is 
easy:


[expr for s in objects if isinstance(s, str) while s.startswith("A")]


Why should we prohibit expressing this, and instead write it as this?

[expr for s in objects while (s.startswith("A")) if isinstance(s, str) else 
True) if isinstance(s, str)]


Or split into multiple comprehensions?

[expr for s in [obj for obj in objects if isinstance(obj, str)] while 
s.startswith("A")]


> (expr for var in seq while loop_cond if filter_cond)
> 
> <=>
> 
> for var in seq:
> if loop_cond:
> if filter_cond:
> yield expr
> else:
> break


We can still expand the clauses if they are presented in the opposite 
order:

(expr for var in seq if filter_cond while loop_cond)

<=>

for var in seq:
if filter_cond:
if loop_cond:
yield expr
else:
break

There's no need to prohibit that. It is meaningful and useful and just 
because somebody might accidentally fail to exit an infinite loop is no 
reason to ban this.


> This is another good reason why a termination condition would need to
> be checked before the filter condition rather than either after it, or
> only when the filter condition was true.

Why is this a problem that needs solving?

Who is to say that an infinite generator expression isn't exactly what 
the programmer wants? If the halting condition is not true, the 
generator expression will either keep going until the iterator is 
exhausted, or it will be an infinite generator just like the 
unprocessed, unfiltered source iterator. This is not necessarily a 
problem.




-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 11 August 2017 at 06:53, Spencer Brown  wrote:
> The logical solution to me is to allow any order of while and if, and follow
> the same 'rule' as multiple for loops - just nest/test those in that order.
> Then you can have whatever priority you need. One question though is how
> this should handle multiple loops - break all of them, or just the current
> one?

This is why I think a for-while construct in comprehensions would
really only make sense in combination with a *statement* level
for-while construct, as the problem we have is:

- a termination condition can't readily use "if" (even in combination
with "break") because that's visually and syntactically ambiguous with
a filter condition
- a naive translation of a "while" based syntax makes it look like a
nested *non-terminating* loop

Both of those problems may be resolved if a "for-while" loop exists as
a top level looping construct that can terminate based on *either* an
iterator being exhausted *or* a condition becoming false.

The question then becomes whether or not a "for-while" loop is
actually useful enough to be added as a general construct, given that
we already have "if not condition: break" as a way of modeling a loop
ending early because a condition became false.

One way to gather evidence on that front would be to survey the
standard library for places where we use "break", and see if any of
them would be more readable given a for-while construct, whether as a
statement, or as part of the comprehension syntax. (Note: I'm not
interested enough in the idea to do that evidence gathering myself,
I'm just pointing it out in case anyone is curious enough to take the
time to collect those details)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Thu, Aug 10, 2017 at 01:25:24PM -0700, Chris Barker wrote:
> On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:
> 
> 
> >  Also, there's a potential issue
> > here - consider
> >
> > [expr for var in even_numbers() if is_odd(var) while var < 100]
> >
> > This is an infinite loop, even though it has a finite termination
> > condition (var < 100), because we only test the termination condition
> > if var is odd, which it never will be.

I'm not sure why Paul thinks this is an issue. There are plenty of ways 
to accidentally write an infinite loop in a comprehension, or a for 
loop, already:

[expr for var in even_numbers()]

will do it, if even_numbers is unexpectedly an infinite iterator. Or you 
could write:

for num in even_numbers():
if is_odd(num) and num > 100:
break

No loop syntax, whether it is functional style (takewhile, map, etc.), 
comprehension, or traditional style for loops, enables the programmer 
to avoid thinking about what they write.


> why is the termination only tested if teh if clause is True? Could then not
> be processed in parallel? or the while first

Because we're following the standard Python rule of left-to-right 
execution. The while clause is tested only if the if clause is true 
because it follows the if clause.

I think that there's an argument to be made for the rule:

We can have `if` in a comprehension, or `while`, but not both

in order to limit complexity. Analogy: 

(1) we intentionally limit the decorator @ syntax to a subset of 
expressions;

(2) likewise we intentionally allow (but don't encourage) monkey-
patching of Python classes only, not built-ins.

Just because we *can* allow arbitrary code combinations doesn't mean we 
*must*. We have a choice to say:

"No, you cannot mix `if` and `when` in the same comprehension. 
Why? Because we say so. Because it is confusing if you do."

I'd be okay with that rule.

But if we decide to allow arbitrary combinations of for/if/while in 
comprehensions, then I think we must keep the same left-to-right rule we 
have now. Currently we process multiple for/if clauses left-to-right:

[expr for x in a if cond for y in b]

is equivalent to:

for x in a:
if cond:
for y in b:
expr

rather than moving the `if` to the end. If you want it at the end, put 
it there yourself. Adding `while` shouldn't change that. It would be 
crazy-complicated to have a rule:

"the presence of a while means the comprehension is 
processed in parallel"

or

"all the while clauses are processed before (after?)
the if clauses, regardless of their order of appearance."


> so maybe better to do:
> 
> [expr for var in even_numbers() while var < 100 if is_odd(var)]

Well sure, that's the *correct* way to write the code:

for var in even_numbers():
if not (var < 100): break
if is_odd(var):
results.append(expr)

(for some definition of "correct" -- this is clearly an expensive way to 
generate an empty list.)

But in general one might wish to test the if or the while in either 
order.


> Maybe it's just me, but I would certainly expect the while to have
> precedence.

Does that apply to these idioms as well?

while cond:
if flag:
...

versus:

if flag:
while cond:
...


I would not expect them to be the same, and nor would I expect these to 
be the same:

[expr for x in seq if flag while cond]

[expr for x in seq while cond if flag]


> I guess I think of it like this:
> 
> "if" is providing a filtering mechanism
> 
> "while" is providing a termination mechanism
> 
>  -- is there a use case anyone can think of when they would want the while
> to be applied to the list AFTER filtering?

[process(n) for n in numbers while n > 0 if is_odd(n)] 

Halt on the first zero or negative number, regardless of whether it is 
even or odd, but process only odd numbers.



Paul: 
> > Obviously, this is a contrived example. And certainly "don't do 
> > that, then" is a valid response. But my instinct is that people are 
> > going to get this wrong - *especially* in a maintenance environment.

That's the argument for limiting comprehensions to either `if` or 
`while` but not both. And I actually would be okay with that -- 
especially if we leave open the possibility of relaxing the prohibition 
in the future.

But personally, I think that's under-estimating the ability of 
programmers to reason about loops. Of course a comprehension with 
multiple for/if/while clauses is hard to reason about, and we shouldn't 
*encourage* them, but we don't prohibit multiple for/if clauses. Why 
should `while` be held to a higher standard? If we allow people to shoot 
themselves in the foot by writing complex list comprehensions with ten 
`for` loops and seven `if` clauses, why should we baulk at allowing them 
a `while` clause as well?



-- 
Steve
___
Python-ideas mailing list
Python-

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 11 August 2017 at 01:39, Paul Moore  wrote:
> On 10 August 2017 at 14:42, Steven D'Aprano  wrote:
>> I don't think it is confusing. Regardless of the implementation, the
>> meaning of:
>>
>> [expression for x in sequence while condition]
>>
>> should (I believe) be obvious to anyone who already groks comprehension
>> syntax. The mapping to a for-loop is admittedly a tad more complex:
>>
>> result = []
>> for x in sequence:
>> if not condition: break
>> result.append(expression)
>>
>> but I'm yet to meet anyone who routinely and regularly reads
>> comprehensions by converting them to for loops like that. And if they
>> did, all they need do is mentally map "while condition" to "if not
>> condition: break" and it should all Just Work™.
>
> The hard part is the interaction between if and while.
>
> Consider (expr for var in seq if cond1 while cond2):
>
> This means:
>
> for var in seq:
> if cond1:
> if not cond2: break
> yield expr
>
> Note that unlike all other comprehension clauses (for and if) while
> doesn't introduce a new level of nesting. That's an inconsistency, and
> while it's minor, it would need clarifying (my original draft of this
> email was a mess, because I misinterpreted how if and while would
> interact, precisely over this point).

This is actually how I came to the conclusion that if we were ever to
do something like this, the termination condition would need to go
*before* the filter condition:

(expr for var in seq while loop_cond if filter_cond)

<=>

for var in seq:
if loop_cond:
if filter_cond:
yield expr
else:
break

With the clauses in that order, the "while" keyword effectively
operates as "if-else-break" the same way it does in a regular while
loop, and could potentially be introduced as a modifying clause on
regular for loops at the same time.

One of the neat things the latter would allow is to make it even
easier to introduce a diagnostic loop counter into while loops:

while condition:
...

could become:

for iteration in itertools.count(1) while condition:
...

rather than having to implement a manually incremented loop counter
the way you do today.

> Also, there's a potential issue
> here - consider
>
> [expr for var in even_numbers() if is_odd(var) while var < 100]
>
> This is an infinite loop, even though it has a finite termination
> condition (var < 100), because we only test the termination condition
> if var is odd, which it never will be.

This is another good reason why a termination condition would need to
be checked before the filter condition rather than either after it, or
only when the filter condition was true.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Nick Coghlan]

On 11 August 2017 at 01:39, Soni L.  wrote:
> I'm pretty sure I read somewhere that lambdas and generators share their
> syntax, and that syntax is already a subset of python syntax. Would it be
> too hard to expose that with a "simplified AST" API?

We already do, via the "mode" argument to the compile builtin and to ast.parse:

>>> ast.dump(ast.parse("1000 <= x < 100", mode="eval"))
"Expression(body=Compare(left=Num(n=1000), ops=[LtE(), Lt()],
comparators=[Name(id='x', ctx=Load()), Num(n=100)]))"

>>> ast.parse("import sys", mode="eval")
Traceback (most recent call last):
 File "", line 1, in 
 File "/usr/lib64/python3.6/ast.py", line 35, in parse
   return compile(source, filename, mode, PyCF_ONLY_AST)
 File "", line 1
   import sys
^
SyntaxError: invalid syntax

It's a large part of the reason why passing strings around has so far
qualified as "good enough" - providing dedicated syntax for it doesn't
actually increase the language's expressiveness all that much, it just
has the potential to make static analysis easier by eagerly rendering
to an AST rather than having that be handled by the function receiving
the argument.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PyPI JSON Metadata Standardization for Mirrors

[Wes Turner]

On Wednesday, August 9, 2017, Cooper Ry Lees  wrote:

> Hi all,
>
> First time emailer, so please be kind. Also, if this is not the right
> mailing list for PyPA talk, I apologize. Please point me in the right
> direction if so.
>

Here are some notes re: changing metadata:

https://github.com/pypa/interoperability-peps/issues/31

https://www.google.com/search?q=pep426jsonld

Towards JSONLD is the best approach, I think. So, that means it would be
best to, if you need to add additional metadata (?) and must key things,
also copy the key into an object:

{"thing1": {"@id": "thing1", "url": "..."}}

Instead of just:

{"thing1": {"url": "..."}}

https://github.com/pypa/interoperability-peps/issues/31#issuecomment-233195564


>  The main reason I have emailed here is I believe it may be PEP time to
> standardize the JSON metadata that PyPI makes available, like what was done
> for the `'simple API` described in PEP503.
>
> I've been doing a bit of work on `bandersnatch` (I didn't name it), which
> is a PEP 381 mirroring package and wanted to enhance it to also mirror the
> handy JSON metadata PyPI generates and makes available @
> https://pypi.python.org/pypi/PKG_NAME/json.
>
> I've done a PR on bandersnatch as a POC that mirrors both the PyPI
> directory structure (URL/pypi/PKG_NAME/json) and created a standardizable
> URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some
> other proxy). I'm also contemplating naming the directory 'metadata' rather
> than JSON so if some new hotness / we want to change the format down the
> line we're not stuck with json as the dirname. This PR can be found here:
> https://bitbucket.org/pypa/bandersnatch/pull-requests/33/save-json-
> metadata-to-mirror
>
> My main use case is to write a very simple async 'verifier' tool that will
> crawl all the JSON files and then ensure the packages directory on each of
> my internal mirrors (I have a mirror per region / datacenter) have all the
> files they should. I sync centrally (to save resource on the PyPI
> infrastructure) and then rsync out all the diffs to each region /
> datacenter, and under some failure scenarios I could miss a file or many.
> So I feel using JSON pulled down from the authoritative source will allow
> an async job to verify the MD5 of all the package files on each mirror.
>
> What are peoples thoughts here? Is it worth a PEP similar to PEP503 going
> forward? Can people enhance / share some thoughts on this idea.
>

Here are some notes on making this more efficient:

"Add API endpoint to get latest version of all projects"
https://github.com/pypa/warehouse/issues/347


... To http://markmail.org/search/?q=list:org.python.distutils-sig .


>
> Thanks,
> Cooper Lees
> m...@cooperlees.com 
> https://cooperlees.com/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Thu, Aug 10, 2017 at 1:53 PM, Spencer Brown  wrote:

> The logical solution to me is to allow any order of while and if, and
> follow the same 'rule' as multiple for loops - just nest/test those in that
> order.
>

Actually, I think it would be better to only allow one order, and have the
"while" always teeted first -- which may mean it should be placed first for
clarity.


> Then you can have whatever priority you need. One question though is how
> this should handle multiple loops - break all of them, or just the current
> one?
>

just the current one, just like a "break", or for that matter, a nested
while...

-CHB





> - Spencer Brown
>
> On 11 Aug 2017, at 6:27 am, Chris Barker  wrote:
>
> On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:
>
>
>>  Also, there's a potential issue
>> here - consider
>>
>> [expr for var in even_numbers() if is_odd(var) while var < 100]
>>
>> This is an infinite loop, even though it has a finite termination
>> condition (var < 100), because we only test the termination condition
>> if var is odd, which it never will be.
>>
>
> why is the termination only tested if teh if clause is True? Could then
> not be processed in parallel? or the while first
>
> so maybe better to do:
>
> [expr for var in even_numbers() while var < 100 if is_odd(var)]
>
> Maybe it's just me, but I would certainly expect the while to have
> precedence.
>
> I guess I think of it like this:
>
> "if" is providing a filtering mechanism
>
> "while" is providing a termination mechanism
>
>  -- is there a use case anyone can think of when they would want the while
> to be applied to the list AFTER filtering?
>
> Obviously, this is a contrived example. And certainly "don't do that,
>> then" is a valid response. But my instinct is that people are going to
>> get this wrong - *especially* in a maintenance environment.
>
>
> sure, but would there be an issue if teh while were given precedence?
>
> Overall, I agree with Steven's point. It seems pretty obvious what the
>> intention is, and while it's probably possible to construct examples
>> that are somewhat unclear,
>>
>> 1. The mechanical rule gives an explicit meaning
>> 2. People shouldn't be writing such complex comprehensions, so if the
>> rule doesn't give what they expect, they can always rewrite the code
>> with an explicit (and clearer) loop.
>>
>
> me too -- a direct translation to a for loop isn't necessary to understand
> how it works.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Spencer Brown]

The logical solution to me is to allow any order of while and if, and follow 
the same 'rule' as multiple for loops - just nest/test those in that order. 
Then you can have whatever priority you need. One question though is how this 
should handle multiple loops - break all of them, or just the current one?

- Spencer Brown

On 11 Aug 2017, at 6:27 am, Chris Barker 
mailto:chris.bar...@noaa.gov>> wrote:

On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore 
mailto:p.f.mo...@gmail.com>> wrote:

 Also, there's a potential issue
here - consider

[expr for var in even_numbers() if is_odd(var) while var < 100]

This is an infinite loop, even though it has a finite termination
condition (var < 100), because we only test the termination condition
if var is odd, which it never will be.

why is the termination only tested if teh if clause is True? Could then not be 
processed in parallel? or the while first

so maybe better to do:

[expr for var in even_numbers() while var < 100 if is_odd(var)]

Maybe it's just me, but I would certainly expect the while to have precedence.

I guess I think of it like this:

"if" is providing a filtering mechanism

"while" is providing a termination mechanism

 -- is there a use case anyone can think of when they would want the while to 
be applied to the list AFTER filtering?

Obviously, this is a contrived example. And certainly "don't do that,
then" is a valid response. But my instinct is that people are going to
get this wrong - *especially* in a maintenance environment.

sure, but would there be an issue if teh while were given precedence?

Overall, I agree with Steven's point. It seems pretty obvious what the
intention is, and while it's probably possible to construct examples
that are somewhat unclear,

1. The mechanical rule gives an explicit meaning
2. People shouldn't be writing such complex comprehensions, so if the
rule doesn't give what they expect, they can always rewrite the code
with an explicit (and clearer) loop.

me too -- a direct translation to a for loop isn't necessary to understand how 
it works.

-CHB


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Paul Moore]

On 10 August 2017 at 21:25, Chris Barker  wrote:
> On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:
>
>>
>>  Also, there's a potential issue
>> here - consider
>>
>> [expr for var in even_numbers() if is_odd(var) while var < 100]
>>
>> This is an infinite loop, even though it has a finite termination
>> condition (var < 100), because we only test the termination condition
>> if var is odd, which it never will be.
>
>
> why is the termination only tested if teh if clause is True? Could then not
> be processed in parallel? or the while first

See? That's my point - the "obvious" interpretation stops being
obvious pretty fast...

> so maybe better to do:
>
> [expr for var in even_numbers() while var < 100 if is_odd(var)]

That would work. But I bet people's intuition wouldn't immediately
lead to that fix (or indeed, necessarily incline them to put the
clauses in this order in the first place).

> Maybe it's just me, but I would certainly expect the while to have
> precedence.
>
> I guess I think of it like this:
>
> "if" is providing a filtering mechanism
>
> "while" is providing a termination mechanism
>
>  -- is there a use case anyone can think of when they would want the while
> to be applied to the list AFTER filtering?

Probably not, but when you can have multiple FORs, WHILEs and IFs, in
any order, explaining the behaviour precisely while still preserving
some sense of "filtering comes after termination" is going to be
pretty difficult. [expr for var1 in seq1 if cond1 for var2 in seq2 for
var3 in seq3 if cond2 if cond3] is legal - stupid, but legal. Now add
while clauses randomly in that, and define your expected semantics
clearly so a user (and the compiler!) can determine what the resulting
mess means.

The main benefit of the current "works like a for loop" interpretation
is that it's 100% explicit. Nothing will make a mess like the above
good code, but at least it's well-defined.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Thu, Aug 10, 2017 at 1:03 PM, Terry Reedy  wrote:

> After all, we have "break" in both for and while loops, so clearly there is
>>> the use case...
>>>
>>
> In both cases, we use 'break' to mean break.  If we want to break
> comprehensions, I think we should continue to use 'break' to mean break
> instead of twisting 'while' to mean 'break'.


I was thinking that too.


>> [expression for x in sequence if condition break]


hmm, but if you want to filter, also?

[expression for x in sequence if condition if condition break]

or

[expression for x in sequence if condition break if condition ]

both of those seem more confusing to me than while.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Chris Barker]

On Thu, Aug 10, 2017 at 8:39 AM, Paul Moore  wrote:


>  Also, there's a potential issue
> here - consider
>
> [expr for var in even_numbers() if is_odd(var) while var < 100]
>
> This is an infinite loop, even though it has a finite termination
> condition (var < 100), because we only test the termination condition
> if var is odd, which it never will be.
>

why is the termination only tested if teh if clause is True? Could then not
be processed in parallel? or the while first

so maybe better to do:

[expr for var in even_numbers() while var < 100 if is_odd(var)]

Maybe it's just me, but I would certainly expect the while to have
precedence.

I guess I think of it like this:

"if" is providing a filtering mechanism

"while" is providing a termination mechanism

 -- is there a use case anyone can think of when they would want the while
to be applied to the list AFTER filtering?

Obviously, this is a contrived example. And certainly "don't do that,
> then" is a valid response. But my instinct is that people are going to
> get this wrong - *especially* in a maintenance environment.


sure, but would there be an issue if teh while were given precedence?

Overall, I agree with Steven's point. It seems pretty obvious what the
> intention is, and while it's probably possible to construct examples
> that are somewhat unclear,
>
> 1. The mechanical rule gives an explicit meaning
> 2. People shouldn't be writing such complex comprehensions, so if the
> rule doesn't give what they expect, they can always rewrite the code
> with an explicit (and clearer) loop.
>

me too -- a direct translation to a for loop isn't necessary to understand
how it works.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Terry Reedy]

On 8/10/2017 9:42 AM, Steven D'Aprano wrote:

On Wed, Aug 09, 2017 at 01:23:28PM -0700, Chris Barker wrote:


I can't recall the use case(s) at the moment, but I have definitely wanted
a way to break out of a comprehension -- and not always with infinite
iterators.

After all, we have "break" in both for and while loops, so clearly there is
the use case...


In both cases, we use 'break' to mean break.  If we want to break 
comprehensions, I think we should continue to use 'break' to mean break 
instead of twisting 'while' to mean 'break'.



[expression for x in sequence while condition]

should (I believe) be obvious to anyone who already groks comprehension
syntax. The mapping to a for-loop is admittedly a tad more complex:

result = []
for x in sequence:
 if not condition: break
 result.append(expression)


This is the same as

result = []
for x in sequence:
if condition:
result.append(expression)
else:
break

which could be written

[expression for x in sequence if condition break]

--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PyPI JSON Metadata Standardization for Mirrors

[Brett Cannon]

The proper list for this would be distutils-sig as that's where
packaging-related discussions typically occur.

On Wed, 9 Aug 2017 at 21:22 Cooper Ry Lees  wrote:

> Hi all,
>
> First time emailer, so please be kind. Also, if this is not the right
> mailing list for PyPA talk, I apologize. Please point me in the right
> direction if so. The main reason I have emailed here is I believe it may be
> PEP time to standardize the JSON metadata that PyPI makes available, like
> what was done for the `'simple API` described in PEP503.
>
> I've been doing a bit of work on `bandersnatch` (I didn't name it), which
> is a PEP 381 mirroring package and wanted to enhance it to also mirror the
> handy JSON metadata PyPI generates and makes available @
> https://pypi.python.org/pypi/PKG_NAME/json.
>
> I've done a PR on bandersnatch as a POC that mirrors both the PyPI
> directory structure (URL/pypi/PKG_NAME/json) and created a standardizable
> URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some
> other proxy). I'm also contemplating naming the directory 'metadata' rather
> than JSON so if some new hotness / we want to change the format down the
> line we're not stuck with json as the dirname. This PR can be found here:
> https://bitbucket.org/pypa/bandersnatch/pull-requests/33/save-json-metadata-to-mirror
>
> My main use case is to write a very simple async 'verifier' tool that will
> crawl all the JSON files and then ensure the packages directory on each of
> my internal mirrors (I have a mirror per region / datacenter) have all the
> files they should. I sync centrally (to save resource on the PyPI
> infrastructure) and then rsync out all the diffs to each region /
> datacenter, and under some failure scenarios I could miss a file or many.
> So I feel using JSON pulled down from the authoritative source will allow
> an async job to verify the MD5 of all the package files on each mirror.
>
> What are peoples thoughts here? Is it worth a PEP similar to PEP503 going
> forward? Can people enhance / share some thoughts on this idea.
>
> Thanks,
> Cooper Lees
> m...@cooperlees.com 
> https://cooperlees.com/
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Soni L.]

On 2017-08-10 01:11 PM, Steven D'Aprano wrote:

On Thu, Aug 10, 2017 at 12:39:34PM -0300, Soni L. wrote:


I'm pretty sure I read somewhere that lambdas and generators share their
syntax, and that syntax is already a subset of python syntax. Would it
be too hard to expose that with a "simplified AST" API?

I don't understand what you mean by this.

The syntax for lambda is (roughly):

  lambda parameter-list : expression

The syntax for generators is (again, roughly):

  def name ( parameter-list ) :
  suite-containing-yield


Obviously the generator suite can contain expressions, and both have a
parameter-list. What shared syntax are you referring to, and how is it
relevant?

Or are you referring to generator expressions, rather than generators?

  ( expression for target in expression ... )

Obviously a Python expression is a Python expression, wherever it is, so
a lambda can contain generator expressions, and generator expressions
can contain lambdas...

And what do you mean by "simplified AST" API? I'm afraid your comment is
too abstract for me to understand.



Yes, see, both are expressions. Expression AST is a subset of python 
AST, so it's a simplified form of the python AST.






___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Thu, Aug 10, 2017 at 12:39:34PM -0300, Soni L. wrote:

> I'm pretty sure I read somewhere that lambdas and generators share their 
> syntax, and that syntax is already a subset of python syntax. Would it 
> be too hard to expose that with a "simplified AST" API?

I don't understand what you mean by this.

The syntax for lambda is (roughly):

 lambda parameter-list : expression

The syntax for generators is (again, roughly):

 def name ( parameter-list ) :
 suite-containing-yield


Obviously the generator suite can contain expressions, and both have a 
parameter-list. What shared syntax are you referring to, and how is it 
relevant?

Or are you referring to generator expressions, rather than generators?

 ( expression for target in expression ... )

Obviously a Python expression is a Python expression, wherever it is, so 
a lambda can contain generator expressions, and generator expressions 
can contain lambdas... 

And what do you mean by "simplified AST" API? I'm afraid your comment is 
too abstract for me to understand.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Soni L.]

On 2017-08-10 01:43 AM, Nick Coghlan wrote:

On 10 August 2017 at 01:49, Soni L.  wrote:

On 2017-08-09 11:54 AM, Nick Coghlan wrote:

Right, I was separating the original request to make "{x for x in
integers if 1000 <= x < 100}" work into the concrete proposal to
make exactly *that* syntax work (which I don't think is feasible), and
the slightly more general notion of offering a more math-like syntax
that allows finite sets to be built from infinite iterators by
defining a termination condition in addition to a filter condition.

Ok. A concrete proposal would give a read-only 'filter' argument to the
iterator somehow, which represents some form of simplified AST of the
condition.

So e.g. {x for x in integers if (lambda v: 1000 <= v < 100)(x)} would
never complete, but {x for x in integers if 1000 <= x < 100} would. (But
perhaps lambda objects should include an AST attribute... Having it for
normal functions would introduce too much overhead tho, and then it would no
longer be a simplified AST, but rather a complete python AST, which we don't
want.)

There have been a variety of different "thunking" proposals over the
years, but they've all foundered on the question of what the
*primitive* quoted form should look like, and how the thunks should
subsequently be executed.

For cases like this, where integration with Python's name resolution
mechanism isn't actually required, folks have ended up just using
strings, where the only downside is the fact that syntax highlighters
and other static analysers don't know that the contents are supposed
to be valid Python code. In a case like this, that might look like:

 {x for x in integers.build_set("1000 <= x < 100")}

As with regexes, the cost of dynamically parsing such strings can then
be amortised at runtime through the use of an appropriate caching
strategy.


I'm pretty sure I read somewhere that lambdas and generators share their 
syntax, and that syntax is already a subset of python syntax. Would it 
be too hard to expose that with a "simplified AST" API?




Cheers,
Nick.



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Paul Moore]

On 10 August 2017 at 14:42, Steven D'Aprano  wrote:
> I don't think it is confusing. Regardless of the implementation, the
> meaning of:
>
> [expression for x in sequence while condition]
>
> should (I believe) be obvious to anyone who already groks comprehension
> syntax. The mapping to a for-loop is admittedly a tad more complex:
>
> result = []
> for x in sequence:
> if not condition: break
> result.append(expression)
>
> but I'm yet to meet anyone who routinely and regularly reads
> comprehensions by converting them to for loops like that. And if they
> did, all they need do is mentally map "while condition" to "if not
> condition: break" and it should all Just Work™.

The hard part is the interaction between if and while.

Consider (expr for var in seq if cond1 while cond2):

This means:

for var in seq:
if cond1:
if not cond2: break
yield expr

Note that unlike all other comprehension clauses (for and if) while
doesn't introduce a new level of nesting. That's an inconsistency, and
while it's minor, it would need clarifying (my original draft of this
email was a mess, because I misinterpreted how if and while would
interact, precisely over this point). Also, there's a potential issue
here - consider

[expr for var in even_numbers() if is_odd(var) while var < 100]

This is an infinite loop, even though it has a finite termination
condition (var < 100), because we only test the termination condition
if var is odd, which it never will be.

Obviously, this is a contrived example. And certainly "don't do that,
then" is a valid response. But my instinct is that people are going to
get this wrong - *especially* in a maintenance environment. That
example could have started off being "for var in count(0)" and then
someone realised they could "optimise" it by omitting odd numbers,
introducing the bug in the process. (And I'm sure real life code could
come up with much subtler examples ;-))

Overall, I agree with Steven's point. It seems pretty obvious what the
intention is, and while it's probably possible to construct examples
that are somewhat unclear,

1. The mechanical rule gives an explicit meaning
2. People shouldn't be writing such complex comprehensions, so if the
rule doesn't give what they expect, they can always rewrite the code
with an explicit (and clearer) loop.

But while I think this says that the above interpretation of while is
the only sensible one, and in general other approaches are unlikely to
be as natural, I *don't* think that it unequivocally says that
allowing while is a good thing. It may still be better to omit it, and
force people to state their intent explicitly (albeit a bit more
verbosely).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Thu, Aug 10, 2017 at 12:54:57AM +1000, Nick Coghlan wrote:

Guido wrote:
> > I haven't come around to this yet. It looks like it will make explaining
> > comprehensions more complex, since the translation of "while X" into "if not
> > X: break" feels less direct than the translations of "for x in xs" or "if
> > pred(x)". (In particular, your proposal seems to require more experience
> > with mentally translating loops and conditions into jumps -- most regulars
> > of this forum do that for a living, but I doubt it's second nature for the
> > OP.)
>
> Yeah, if we ever did add something like this, I suspect a translation
> using takewhile would potentially be easier for at least some users to
> understand than the one to a break condition:

"Some users"? Sure, why not? There's probably somebody out there who 
understands takewhile, but if so, I don't know who they are :-)

I always have to look at the docs for takewhile to remind myself whether 
it drops items ("takes them away") while the condition is true, or 
yields items ("gives items") while the condition is true.


> {x for x in itertools.count(0) if 1000 <= x while x < 100}
> 
> <=>
> 
> x = set()
> for x in itertools.count(0):
> if 1000 <= x:
> set.add(x)
> # If you've never used the loop-and-a-half idiom, it's
> # not obvious why "while " means "if not : break"
> if not x < 100:
> break

I'd like to take issue with that "not obvious" comment. I think that 
anyone who knows while loops knows that the loop exits when the 
condition becomes false. That's exactly the behaviour we get for the 
(hypothetical) [expr for x in seq while condition] syntax: when the 
condition is false, the loop and hence the comprehension, exits.

For such simple cases, there's no need to think about "loop and a half". 
The obvious explanation is that the loop exits when the while condition 
fails.

Based on my experience with beginners on the tutor mailing list, and 
elsewhere, I think there's a definite learning "hump" to get over before 
people grok even the trivial case of

[expression for x in sequence]

but once they do, then adding an "if" clause is obvious, and I expect 
that the same will apply to "when".

Once you move beyond the simple case of a single for and no more than a 
single if (or while), I don't think there's *anything* obvious about 
comprehension syntax at all, while clause or no while clause. Holding 
the while clause to a standard that comprehensions already fail (in my 
opinion) is unfair:

[expression for x in seq1 for y in seq2 if pred1 for z in seq3 
if pred2 if pred3 if pred4 for w in seq4 while condition for v in seq5]

I don't think it's the "while" that tips that over the edge, 
readability-wise :-)


In any case, I think we're all guessing whether or not people will 
understand the "while condition" syntax. So I've done an informal survey 
on the Python-Ideas list, and once folks have had a day or so to answer 
I'll report what they say. It's not a truly scientific UI test, but it's 
the best I can do.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Generator syntax hooks?

[Steven D'Aprano]

On Wed, Aug 09, 2017 at 01:23:28PM -0700, Chris Barker wrote:

> I can't recall the use case(s) at the moment, but I have definitely wanted
> a way to break out of a comprehension -- and not always with infinite
> iterators.
>
> After all, we have "break" in both for and while loops, so clearly there is
> the use case...

Indeed :-)


> If someone comes up with a clean and not confusing (and general purpose)
> syntax, I think it would be very useful.

We used to be able to (ab)use StopIteration to do this:

def Break():
raise StopIteration

# generator expressions only, not list comprehensions
result = (expression for x in sequence if condition or Break())

but I believe that loophole has been closed in 3.6.


Comprehensions in Clojure have this feature:

http://clojuredocs.org/clojure_core/clojure.core/for

Clojure uses "when" where Python uses "if", giving:

;; :when continues through the collection even if some have the
;; condition evaluate to false, like filter
user=> (for [x (range 3 33 2) :when (prime? x)]
 x)
(3 5 7 11 13 17 19 23 29 31)

;; :while stops at the first collection element that evaluates to
;; false, like take-while
user=> (for [x (range 3 33 2) :while (prime? x)]
 x)
(3 5 7)


Translating into Python:

[x for x in range(3, 33, 2) if is_prime(x)]

[x for x in range(3, 33, 2) while is_prime(x)]  # hypothetical syntax

I don't think it is confusing. Regardless of the implementation, the 
meaning of:

[expression for x in sequence while condition]

should (I believe) be obvious to anyone who already groks comprehension 
syntax. The mapping to a for-loop is admittedly a tad more complex:

result = []
for x in sequence:
if not condition: break
result.append(expression)

but I'm yet to meet anyone who routinely and regularly reads 
comprehensions by converting them to for loops like that. And if they 
did, all they need do is mentally map "while condition" to "if not 
condition: break" and it should all Just Work™.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/