Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Nick Coghlan]

On 21 October 2016 at 05:09, Random832  wrote:
> On Tue, Oct 18, 2016, at 02:10, Nick Coghlan wrote:
>> Hi, I contributed the current list comprehension implementation (when
>> refactoring it for Python 3 to avoid leaking the iteration variable,
>> as requested in PEP 3100 [1]), and "comprehensions are syntactic sugar
>> for a series of nested for and if statements" is precisely my
>> understanding of how they work, and what they mean.
>
> But it's simply not true. It has never been true and it will never be
> true. Something is not "syntactic sugar" if it doesn't compile to the
> exact same sequence of operations as the thing it is supposedly
> syntactic sugar for.

We don't need to guess about this, since we can consult the language
reference and see how comprehension semantics are specified for
language implementors:
https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries

Firstly, container displays are broken out into two distinct kinds:

For constructing a list, a set or a dictionary Python provides
special syntax called “displays”, each of them in two flavors:

- either the container contents are listed explicitly, or
- they are computed via a set of looping and filtering
instructions, called a comprehension.

Secondly, the meaning of the clauses in comprehensions is spelled out
a little further down:

The comprehension consists of a single expression followed by at
least one for clause and zero or more for or if clauses. In this case,
the elements of the new container are those that would be produced by
considering each of the for or if clauses a block, nesting from left
to right, and evaluating the expression to produce an element each
time the innermost block is reached.

We can also go back and read the design PEPs that added these features
to the language:

* List comprehensions: https://www.python.org/dev/peps/pep-0202/
* Generator expressions: https://www.python.org/dev/peps/pep-0289/

PEP 202 defined the syntax in terms of its proposed behaviour rather
than a syntactic, with the only reference to the nesting equivalence
being this BDFL pronouncement:

- The form [... for x... for y...] nests, with the last index
  varying fastest, just like nested for loops.

PEP 289, by contrast, fully spells out the implied generator
definition that was used to guide the implementation of generator
expressions in the code generator:

g = (tgtexp  for var1 in exp1 if exp2 for var2 in exp3 if exp4)

is equivalent to:

def __gen(bound_exp):
for var1 in bound_exp:
if exp2:
for var2 in exp3:
if exp4:
yield tgtexp
g = __gen(iter(exp1))
del __gen

When I implemented the comprehension index variable hiding for Python
3.0, the final version was the one where I blended those two
definitions to end up with the situation where:

data = [(var1, var2)  for var1 in exp1 if exp2 for var2 in exp3 if exp4)]

is now equivalent to:

def __comprehension(bound_exp):
__hidden_var = []
for var1 in bound_exp:
if exp2:
for var2 in exp3:
if exp4:
__hidden_var.append((var1, var2))
return __hidden_var
data = __comprehension(iter(exp1))
del __comprehension

While it's pretty dated now (I wrote it circa 2.5 as part of a draft
book manuscript that was never published), if you'd like to learn more
about this, you may want to check out the section on "Initialising
standard containers" in
http://svn.python.org/view/sandbox/trunk/userref/ODF/Chapter02_StatementsAndExpressions.odt

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Chris Angelico]

On Fri, Oct 21, 2016 at 6:09 AM, Random832  wrote:
> On Tue, Oct 18, 2016, at 02:10, Nick Coghlan wrote:
>> Hi, I contributed the current list comprehension implementation (when
>> refactoring it for Python 3 to avoid leaking the iteration variable,
>> as requested in PEP 3100 [1]), and "comprehensions are syntactic sugar
>> for a series of nested for and if statements" is precisely my
>> understanding of how they work, and what they mean.
>
> But it's simply not true. It has never been true and it will never be
> true. Something is not "syntactic sugar" if it doesn't compile to the
> exact same sequence of operations as the thing it is supposedly
> syntactic sugar for. It's a useful teaching tool (though you've
> eventually got to teach the differences), but claiming that it's
> "syntactic sugar" - and "non-negotiably" so, at that - implies that it
> is a literal transformation.

But it is. There are two caveats to the transformation: firstly, it's
done in a nested function (as of Py3), and secondly, the core
operations are done with direct opcodes rather than looking up the
".append" method; but other than that, yes, it's exactly the same.
Here's the disassembly (in 2.7, to avoid the indirection of the nested
function):

>>> def f1(x):
... return [ord(ch) for ch in x]
...
>>> dis.dis(f1)
  2   0 BUILD_LIST   0
  3 LOAD_FAST0 (x)
  6 GET_ITER
>>7 FOR_ITER18 (to 28)
 10 STORE_FAST   1 (ch)
 13 LOAD_GLOBAL  0 (ord)
 16 LOAD_FAST1 (ch)
 19 CALL_FUNCTION1
 22 LIST_APPEND  2
 25 JUMP_ABSOLUTE7
>>   28 RETURN_VALUE
>>> def f2(x):
... ret = []
... for ch in x:
... ret.append(ord(ch))
... return ret
...
>>> dis.dis(f2)
  2   0 BUILD_LIST   0
  3 STORE_FAST   1 (ret)

  3   6 SETUP_LOOP  33 (to 42)
  9 LOAD_FAST0 (x)
 12 GET_ITER
>>   13 FOR_ITER25 (to 41)
 16 STORE_FAST   2 (ch)

  4  19 LOAD_FAST1 (ret)
 22 LOAD_ATTR0 (append)
 25 LOAD_GLOBAL  1 (ord)
 28 LOAD_FAST2 (ch)
 31 CALL_FUNCTION1
 34 CALL_FUNCTION1
 37 POP_TOP
 38 JUMP_ABSOLUTE   13
>>   41 POP_BLOCK

  5 >>   42 LOAD_FAST1 (ret)
 45 RETURN_VALUE
>>>


Okay, so what exactly is going on here? Looks to me like there's some
optimization happening in the list comp, but you can see that the same
code is being emitted. It's not *perfectly* identical, but that's
mainly because CPython doesn't take advantage of the fact that 'ret'
was initialized to a list - it still does the full "look up 'append',
then call it" work. I'm not sure why SETUP_LOOP exists in the full
version and not the comprehension, but I believe it's to do with the
break and continue keywords, which can't happen inside a
comprehension. So, again, it's optimizations that are possible in the
comprehension, but otherwise, the code is identical.

Maybe "syntactic sugar" is pushing it a bit, but there's no
fundamental difference between the two. Imagine if an optimizing
compiler could (a) notice that there's no use of break/continue, and
(b) do some static type analysis to see that 'ret' is always a list
(and not a subclass thereof), and optimize the multi-line version. At
that point, the two forms would look almost, or maybe completely,
identical. So I'd support the "syntactic sugar" label here.

Why is this discussion still on python-ideas? Shouldn't it be on
python-demanding-explanations-for-status-quo by now?

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Random832]

On Tue, Oct 18, 2016, at 02:10, Nick Coghlan wrote:
> Hi, I contributed the current list comprehension implementation (when
> refactoring it for Python 3 to avoid leaking the iteration variable,
> as requested in PEP 3100 [1]), and "comprehensions are syntactic sugar
> for a series of nested for and if statements" is precisely my
> understanding of how they work, and what they mean.

But it's simply not true. It has never been true and it will never be
true. Something is not "syntactic sugar" if it doesn't compile to the
exact same sequence of operations as the thing it is supposedly
syntactic sugar for. It's a useful teaching tool (though you've
eventually got to teach the differences), but claiming that it's
"syntactic sugar" - and "non-negotiably" so, at that - implies that it
is a literal transformation.

I said "limited" in reference to the specific claim - which was not
yours - that since "yield a, b" yields a tuple, "yield *x" [and
therefore (*x for...)] ought to also yield a tuple, and I stand by it.
It's the same kind of simplistic understanding that should lead one to
believe that not only the loop variable but also the "result" temporary
ought to exist after the comprehension is executed.

I was being entirely serious in saying that this is like objecting to
normal unpacking on the grounds that an ordinary list display should be
considered syntactic sugar for an unrolled sequence of append calls. In
both cases, the equivalence is not exact,  and there should be room to
at least discuss things that would merely require an additional rule to
be added (or changed, technically making it "result += [...]" would
cover both cases) to the transformation - a transformation which already
results in three different statements depending on whether it is a list
comprehension, a set comprehension, or a generator expression (and a
fourth if you count dict comprehensions, though that's a different
syntax too) - rather than simply declaring them "not negotiable".

Declaring this "not negotiable" was an incredibly hostile dismissal of
everyone else's position. Especially when what's being proposed wouldn't
invalidate the concept, it would just change the exact details of what
the transformation is. Which is more than can be said for not leaking
the variable.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Sven R. Kunze]

On 18.10.2016 08:23, Greg Ewing wrote:

If it were a namedtuple, for example, you could write

  [*t for t in fulltext_tuples if t.language == 'english']

or

  [x for t in fulltext_tuples if t.language == 'english' for x in t]

The latter is a bit unsatisfying, because we are having to
make up an arbitrary name 'x' to stand for an element of t.
Even though the two elements of t have quite different roles,
we can't use names that reflect those roles.


It's an intriguing idea to use namedtuples but in this case one should 
not over-engineer.


What I dislike most are the names of "fulltext_tuple", "x", "t". If I 
were to use it, I think my coworkers would tar and feather me. ;) This 
is one of the cases where it makes absolutely no sense to invent 
artificial names for the sake of naming. I can name a lot of (internal) 
examples where we tried really hard at inventing named concepts which 
make absolutely no sense half a year later even to those who invented 
them. Repeatedly, in the same part of the code.


Each newly named concept introduces another indirection. Thus, we always 
need to find a middle ground between naming and using language features, 
so I (personally) would be grateful for this particular feature. :)



Because of that, to my eyes the version with * makes it easier
to see what is going on.


That's a very nice phrase: "makes it easier to see what is going on". I 
need to remember that.


Cheers,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Sven R. Kunze]

On 18.10.2016 10:01, Daniel Moisset wrote:
So, for me, this feature is something that could be covered with a 
(new) function with no new syntax required. All you have to learn is 
that instead of [*...] you use flatten(...)


The main motivation is not "hey we need yet another way for xyz" but 
"could we remove that inconsistency?". You wrote "[*...]"; which already 
works. It just that it does not work for all kinds of "..." . I hope 
that makes the motivation clearer.



I for one don't need yet another function or function cascade to make 
things work. "list(itertools.chain.from_iterable(...))" just works fine 
for all kinds of "...".



Cheers,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Paul Moore]

On 18 October 2016 at 07:31, Nick Coghlan  wrote:
>
> Forcing ourselves to come up with a name for the series of values
> produced by the outer iteration then makes that name available as
> documentation of our intent for future readers of the code.

This is a key point, that is frequently missed when people propose new
"shorter" syntax, or constructs that "reduce indentation levels".

Make no mistake, coming up with good names is *hard* (and I have
enormous respect for the (often unrecognised) people who come up with
intuitive APIs and names for functions). So it's very easy to be
tempted by "concise" constructs that offer the option of not naming an
operation. But as someone who spends 99% of his time doing maintenance
programming, be sure that the people who support your code will thank
you for spending time naming your abstractions (and avoiding using
constructs like the one proposed here).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Martti Kühne]

> I feel like I should be honest about something else - I'm always a
> little bit confused by the ordering for comprehensions involving
> multiple clauses. For me, it's the fact that:
> [[a for a in b] for b in ['uvw', 'xyz']] == [['u', 'v', 'w'], ['x', 'y',
> 'z']]
> which makes me want to write:
> [a for a in b for b in ['uvw', 'xyz']]
> but that's an error, and it actually needs to be
> [a for b in ['uvw', 'xyz'] for a in b] == ['u', 'v', 'w', 'x', 'y', 'z']
>
> So when this talk of readability issues comes up and the recommended
> alternative is something that I don't really find readable, it's
> frustrating. To me this proposal is something that would allow for more
> things to be expressed without resorting to multi-loop comprehensions.
>


Thinking about it, though, I ended up exactly where you are now,
except that I then thought about where an item would be known, and it
seemed to me, yes, an item would be more likely known *after* looping
over it rather than before:

[bi for bi in before for before in iterable] # why should "before"
exist before it is looped over?

correctly:
[bi for before in iterable for bi in before]

it doesn't, it should be declared to the right hand side and only the
result is kept over at the left hand edge.

On a same note, with if expressions the picture might look different:

[bi for bi in before for before in iterable if before[0] < 3] # ... is
bi filtered now or not?

correctly:
[bi for before in iterable if before[0] < 3 for bi in before]

it is filtered very clearly this way.

cheers!
mar77i
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Nick Coghlan]

On 18 October 2016 at 13:32, David Mertz  wrote:
> On Mon, Oct 17, 2016 at 7:50 PM, Random832  wrote:
>> I feel like I should be honest about something else - I'm always a
>> little bit confused by the ordering for comprehensions involving
>> multiple clauses.
>
> Me too! I get the order of nested loops in comprehensions wrong about 25% of
> the time.  Then it's a NameError, and I fix it.
>
> This is a lot of why I like a utility function like `flatten()` that is
> pretty much self-documenting.  Perhaps a couple other itertools helpers
> would be nice.

This is also one of the main reasons that named generator expression
pipelines can sometimes be easier to read than nested comprehensions:

incrementally_increasing_ranges = (range(end) for end in itertools.count())
flatten = itertools.chain.from_iterable
incrementally_increasing_cycles = flatten(incrementally_increasing_ranges())

Forcing ourselves to come up with a name for the series of values
produced by the outer iteration then makes that name available as
documentation of our intent for future readers of the code.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Brendan Barnwell]

On 2016-10-17 16:35, Steven D'Aprano wrote:

>and many times I have been irritated by the fact that the
>one-item-per-loop invariant exists.  I'm not sure whether I'm in favor of
>this particular syntax, but I'd like to be able to do the kind of things it
>allows.  But doing them inherently requires breaking the invariant you
>describe.

That last point is incorrect. You already can do the kind of things this
thread is about:

 [*t for t in iterable]  # proposed syntax: flatten

can be written as:

 [x for t in iterable for x in t]


	Right, but by "doing those kinds of things" I mean doing them more in a 
more conise way without an extra level of iteration.  (You can "do 
multiplication" by adding repeatedly, but it's still nice to have 
multiplication as an operation.)


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Nick Coghlan]

On 18 October 2016 at 03:49, Random832  wrote:
> On Mon, Oct 17, 2016, at 13:32, Steven D'Aprano wrote:
>> This isn't a small change: it requires not
>> insignificant changes to people's understanding of what list
>> comprehension syntax means and does.
>
> Only if their understanding is limited to a sequence of tokens that it
> supposedly expands to [except for all the little differences like
> whether a variable actually exists]

Hi, I contributed the current list comprehension implementation (when
refactoring it for Python 3 to avoid leaking the iteration variable,
as requested in PEP 3100 [1]), and "comprehensions are syntactic sugar
for a series of nested for and if statements" is precisely my
understanding of how they work, and what they mean. It is also how
they are frequently explained to new Python users.

Directly insulting me and many of the educators who do so much to
bring new users to Python by calling our understanding of a construct
I implemented (and that you apparently love using) limited, is *not*
doing your cause any favours, and is incredibly inappropriate
behaviour for this list.

Regards,
Nick.

[1] https://www.python.org/dev/peps/pep-3100/#core-language

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[David Mertz]

On Mon, Oct 17, 2016 at 7:50 PM, Random832  wrote:

> On Mon, Oct 17, 2016, at 22:17, David Mertz wrote:
> > > [*range(x) for x in range(4)]
> >
> > As Paul or someone pointed out, that's a fairly odd thing to do.
>
> I agree with the specific example of it being an odd thing to do with
> range, it was just an attempt to illustrate with a concrete example.
>

It's also easy to construct examples where the hypothetical * syntax can't
handle a requirement.  E.g. flatten() with levels>1 (yes, of course you can
find some way to nest more loops or more
comprehensions-within-comprehensions to make it work in some way that still
uses the * by force).

I feel like I should be honest about something else - I'm always a
> little bit confused by the ordering for comprehensions involving
> multiple clauses.


Me too! I get the order of nested loops in comprehensions wrong about 25%
of the time.  Then it's a NameError, and I fix it.

This is a lot of why I like a utility function like `flatten()` that is
pretty much self-documenting.  Perhaps a couple other itertools helpers
would be nice.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[David Mertz]

>
> For a more concrete example:
>
> [*range(x) for x in range(4)]
> [*(),*(0,),*(0,1),*(0,1,2)]
> [0, 0, 1, 0, 1, 2]
>

As Paul or someone pointed out, that's a fairly odd thing to do.  It's the
first time that use case has been mentioned in this thread.  It's true
you've managed to construct something that isn't done by flatten().  I
would have had to think a while to see what you meant by the original if
you haven't provided the intermediate interpretations.

Of course, it's *really simple* to spell that in a natural way with
existing syntax that isn't confusing like yours:

[x for end in range(4) for x in range(end)]

There is no possible way to construct something that would use the proposed
syntax that can't be expressed more naturally with a nested loop... because
it's just confusing syntax sugar for exactly that.

Your example looks like some sort of interview quiz question to see if
someone knows obscure and unusual syntax.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Steven D'Aprano]

On Mon, Oct 17, 2016 at 10:33:32PM +0200, Sven R. Kunze wrote:

> Sorry? You know, I am all for real-world code and I also delivered: 
> https://mail.python.org/pipermail/python-ideas/2016-October/043030.html

Your example shows the proposed:

[*(language, text) for language, text in fulltext_tuples if language == 
'english']

which can be written as:

[x for language, text in fulltext_tuples for x in (language, text) if 
language == 'english']

which is only ten characters longer. To me, though, there's simply no 
nice way of writing this: the repetition of "language, text" reads 
poorly regardless of whether there is a star or no star.

If I were doing this more than once, I'd be strongly inclined to invest 
in a simple helper function to make this more readable:

def filter_and_flatten(language, fulltext):
for lang, text in fulltext:
if lang == language:
yield lang
yield text

filter_and_flatten('english', fulltext_tuples)


In some ways, list comprehensions are a trap: their convenience and ease 
of use for the easy cases lure us into using them when we ought to be 
using a generator. But that's just my opinion.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Paul Moore]

On 17 October 2016 at 21:43, Sven R. Kunze  wrote:
> The statement about "cumbersomeness" was specific to this whole issue. Of
> course, importing feature-rich pieces from the stdlib is really cool. It was
> more the missed ability to do the same with list comprehensions of what is
> possible with list displays today. List displays feature * without importing
> anything fancy from the stdlib.

In your other post you specifically mentioned
itertools.chain.from_iterable. I'd have to agree with you that this
specific name feels clumsy to me as well. But I'd argue for finding a
better name, not replacing the function with syntax :-)

Cheers,
Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Sven R. Kunze]

On 17.10.2016 22:26, Paul Moore wrote:

Certainly having to add an import statement is extra typing. But
terseness was *never* a feature of Python. In many ways, a resistance
to overly terse (I could say "Perl-like") constructs is one of the
defining features of the language - and certainly, it's one that drew
me to Python, and one that I value.

I am completely with you on this one, Paul.

The statement about "cumbersomeness" was specific to this whole issue. 
Of course, importing feature-rich pieces from the stdlib is really cool. 
It was more the missed ability to do the same with list comprehensions 
of what is possible with list displays today. List displays feature * 
without importing anything fancy from the stdlib.


Nevermind, it seems we need to wait longer for this issue to come up 
again and maybe again to solve it eventually.


Best,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Paul Moore]

On 17 October 2016 at 21:30, Random832  wrote:
> On Mon, Oct 17, 2016, at 16:12, Paul Moore wrote:
>> And finally, no-one has even *tried* to explain why we need a third
>> way of expressing this construction. Nick made this point, and
>> basically got told that his condition was too extreme. He essentially
>> got accused of constructing an impossible test. And yet it's an
>> entirely fair test, and one that's applied regularly to proposals -
>> and many *do* pass the test.
>
> As the one who made that accusation, my objection was specifically to
> the word "always" - which was emphasized - and which is something that I
> don't believe is actually a component of the test that is normally
> applied. His words, specifically, were "a compelling argument needs to
> be presented that the new spelling is *always* preferable to the
> existing ones"
>
> List comprehensions themselves aren't even always preferable to loops.

Sigh. And no-one else in this debate has ever used exaggerated language.

I have no idea if Nick would reject an argument that had any
exceptions at all, but I don't think it's unreasonable to ask that
people at least *try* to formulate an argument that demonstrates that
the two existing ways we have are inferior to the proposal. Stating
that you're not even willing to try is hardly productive.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Random832]

On Mon, Oct 17, 2016, at 16:12, Paul Moore wrote:
> And finally, no-one has even *tried* to explain why we need a third
> way of expressing this construction. Nick made this point, and
> basically got told that his condition was too extreme. He essentially
> got accused of constructing an impossible test. And yet it's an
> entirely fair test, and one that's applied regularly to proposals -
> and many *do* pass the test.

As the one who made that accusation, my objection was specifically to
the word "always" - which was emphasized - and which is something that I
don't believe is actually a component of the test that is normally
applied. His words, specifically, were "a compelling argument needs to
be presented that the new spelling is *always* preferable to the
existing ones"

List comprehensions themselves aren't even always preferable to loops.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Paul Moore]

On 17 October 2016 at 21:22, Random832  wrote:
> For a more concrete example:
>
> [*range(x) for x in range(4)]
> [*(),*(0,),*(0,1),*(0,1,2)]
> [0, 0, 1, 0, 1, 2]
>
> There is simply no way to get there by using flatten(range(4)). The only
> way flatten *without* a generator expression can serve the same use
> cases as this proposal is for comprehensions of the *exact* form [*x for
> x in y]. For all other cases you'd need list(flatten(...generator
> expression without star...)).

Do you have a real-world example of needing this?

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Paul Moore]

On 17 October 2016 at 20:35, Sven R. Kunze  wrote:
>  P.S. It's very artificial to assume user are unable to use 'from itertools
> import chain' to try to make chain() seem more cumbersome than it is.
>
> I am sorry but it is cumbersome.

Imports are a fundamental part of Python. How are they "cumbersome"?
Is it cumbersome to have to import sys to get access to argv? To
import re to use regular expressions? To import subprocess to run an
external program?

Importing the features you use (and having an extensive standard
library of tools you might want, but which don't warrant being built
into the language) is, to me, a basic feature of Python.

Certainly having to add an import statement is extra typing. But
terseness was *never* a feature of Python. In many ways, a resistance
to overly terse (I could say "Perl-like") constructs is one of the
defining features of the language - and certainly, it's one that drew
me to Python, and one that I value.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Paul Moore]

On 17 October 2016 at 18:32, Steven D'Aprano  wrote:
> On Mon, Oct 17, 2016 at 12:11:46PM -0400, Random832 wrote:
>
>> Honestly, it goes beyond just being "wrong". The repeated refusal to
>> even acknowledge any equivalence between [...x... for x in [a, b, c]]
>> and [...a..., ...b..., ...c...] truly makes it difficult for me to
>> accept some people's _sincerity_.
>
> While we're talking about people being insincere, how about if you take
> a look at your own comments? This "repeated refusal" that you accuse us
> (opponents of this proposal) of is more of a rhetorical fiction than an
> actual reality. Paul, David and I have all acknowledged the point you
> are trying to make. I won't speak for Paul or David, but speaking for
> myself, it isn't that I don't understand the point you're trying to
> make, but that I do not understand why you think that point is
> meaningful or desirable.

For my part:

1. I've acknowledged that equivalence. As well as the fact that the
proposal (specifically, as explained formally by Greg) is
understandable and a viable possible extension.
2. I don't find the "interpolation" equivalence a *good* way of
interpreting list comprehensions, any more than I think that loops
should be explained by demonstrating how to unroll them.
3. I've even explicitly revised my position on the proposal from -1 to
-0 (although I'm tending back towards -1, if I'm honest...).
4. Whether you choose to believe me or not, I've sincerely tried to
understand the proposal, but I pretty much had to insist on a formal
definition of syntax and semantics before I got an explanation that I
could follow.

However:

1. I'm tired of hearing that the syntax is "obvious". This whole
thread proves otherwise, and I've yet to hear anyone from the
"obvious" side of the debate acknowledge that.
2. Can someone summarise the *other* arguments for the proposal? I'm
genuinely struggling to recall what they are (assuming they exist). It
feels like I'm hearing nothing more than "it's obvious what this does,
it's obvious that it's needed and the people saying it isn't are
wrong". That may well not be the truth, but *it's the impression I'm
getting*. I've tried to take a step back and summarise my side of the
debate a couple of times now. I don't recall seeing anyone doing the
same from the other side (Greg's summarised the proposal, but I don't
recall anyone doing the same with the justification for it).
3. The fact is that the proposed behaviour was *specifically* blocked,
*precisely* because of strong concerns that it would cause readability
issues and only had "mild" support. I'm not hearing any reason to
change that decision (sure, there are a few people here offering
something stronger than "mild" support, but it's only a few voices,
and they are not addressing the readability concerns at all). There
was no suggestion in the PEP that this decision was expected to be
revisited later. Maybe there was an *intention* to do so, but the PEP
didn't state it. I'd suggest that this fact alone implies that the
people proposing this change need to write a new PEP for it, but
honestly I don't think the way the current discussion has gone
suggests that there's any chance of putting together a persuasive PEP,
much less a consensus decision.

And finally, no-one has even *tried* to explain why we need a third
way of expressing this construction. Nick made this point, and
basically got told that his condition was too extreme. He essentially
got accused of constructing an impossible test. And yet it's an
entirely fair test, and one that's applied regularly to proposals -
and many *do* pass the test. It's worth noting here that we have had
no real-world use cases, so the common approach of demonstrating real
code, and showing how the proposal improves it, is not available.
Also, there's no evidence that this is a common need, and so it's not
clear to what extent any sort of special language support is
warranted. We don't (as far as I know, and no-one's provided evidence
otherwise) see people routinely writing workarounds for this
construct. We don't hear of trainers saying that pupils routinely try
to do this, and are surprised when it doesn't work (I'm specifically
talking about students *deducing* this behaviour, not being asked if
they think it's reasonable once explained). These are all arguments
that have been used in the past to justify new syntax (and so reach
Nick's "bar").

And we've had a special-case function (flatten) proposed to cover the
most common cases (taking the approach of the 80-20 rule) - but the
only response to that proposal has been "but it doesn't cover
". If it didn't cover a demonstrably common
real-world problem, that would be a different matter - but anyone can
construct cases that aren't covered by *any* given proposal. That
doesn't prove anything.

I don't see any signs of progress here. And I'm pretty much at the
point where I'm losing interest in having the same points repeated at
me 

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Sven R. Kunze]

On 17.10.2016 20:38, David Mertz wrote:
Under my proposed "more flexible recursion levels" idea, it could even 
be:


  [f(x) for x in flatten(it, levels=3)]

There would simply be NO WAY to get that out of the * comprehension 
syntax at all.  But a decent flatten() function gets all the flexibility.


I see what you are trying to do here and I appreciate it. Just one 
thought from my practical experience: I haven't had a single usage for 
levels > 1. levels==1 is basically * which I have at least one example 
for. Maybe, that relates to the fact that we asked our devs to use names 
(as in attributes or dicts) instead of deeply nested list/tuple structures.


Do you think it would make sense to start a new thread just for the sake 
of readability?



Honestly, it goes beyond just being "wrong". The repeated refusal to
even acknowledge any equivalence between [...x... for x in [a, b, c]]
and [...a..., ...b..., ...c...] truly makes it difficult for me to
accept some people's _sincerity_.


I am absolutely sincere in disliking and finding hard-to-teach this 
novel use of * in comprehensions.


You are consistent at least. You don't teach * in list displays, no 
matter if regular lists or comprehensions. +1


 P.S. It's very artificial to assume user are unable to use 'from 
itertools import chain' to try to make chain() seem more cumbersome 
than it is.


I am sorry but it is cumbersome.


Regards,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[David Mertz]

On Mon, Oct 17, 2016 at 9:11 AM, Random832  wrote:

> Once again, this alleged simplicity relies on the chosen example "x for
> x" rather than "f(x) for x" - this one doesn't even put the use of
> flatten in the right place to be generalized to the more complex cases.
> You'd need list(flatten(f(x) for x in iterable))
>

What you're saying is EXACTLY 180 deg reversed from the truth.  It's
*precisely* because it doesn't need the extra complication that `flatten()`
is more flexible and powerful.  I have no idea what your example is meant
to do, but the actual correspondence is:

  [f(x) for x in flatten(it)]

Under my proposed "more flexible recursion levels" idea, it could even be:

  [f(x) for x in flatten(it, levels=3)]

There would simply be NO WAY to get that out of the * comprehension syntax
at all.  But a decent flatten() function gets all the flexibility.


> Honestly, it goes beyond just being "wrong". The repeated refusal to
> even acknowledge any equivalence between [...x... for x in [a, b, c]]
> and [...a..., ...b..., ...c...] truly makes it difficult for me to
> accept some people's _sincerity_.
>

I am absolutely sincere in disliking and finding hard-to-teach this novel
use of * in comprehensions.

Yours, David...

P.S. It's very artificial to assume user are unable to use 'from itertools
import chain' to try to make chain() seem more cumbersome than it is.
Likewise, I would like flatten() in itertools, but I assume the usual
pattern would be importing the function itself.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Steven D'Aprano]

On Sun, Oct 16, 2016 at 02:34:58PM +0200, Sven R. Kunze wrote:

> As this discussion won't come to an end, I decided to consult my girlfriend.
[...]
> >>> [(i,i,i) for i in range(4)]
> [(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3)]

Did you remember to tell your girlfriend that a critical property of 
the "??? for i in range(4)" construct is that it generates one value per 
loop? It loops four times, so it generates exactly four values (in this 
case, each value is a bracketed term).


> Let's remove these inner parentheses again.
> 
> >>> [*(i,i,i) for i in range(4)]
>   File "", line 1
> SyntaxError: iterable unpacking cannot be used in comprehension

It loops four times, so it must generate four values. What's the star 
supposed to do? Turn four loops into twelve?



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[David Mertz]

Actually, I agree with Marietta. I don't care whatsoever about mocking me,
which was a certain element of it. I have thick skin and am confident in
these conversations.

The part that was probably over the line was mocking children who learn to
program or those who teach them. That's a huge and great job. I know I
would not have the skill to teach children effectively. Adults with
technical expertise are much easier for me.

That said, thank you Mark for your empirical research with a test subject.

Best, David

On Oct 16, 2016 9:39 AM, "Mark Lawrence via Python-ideas" <
python-ideas@python.org> wrote:

> On 16/10/2016 16:41, Mariatta Wijaya wrote:
>
>> Her reaction was hilarious:
>>>
>>> "Whom does he teach? Children?"
>>>
>>
>> I sense mockery in your email, and it does not conform to the PSF code
>> of conduct. Please read the CoC before posting in this mailing list. The
>> link is available at the bottom of every python mailing list
>> email.https://www.python.org/psf/codeofconduct/
>> 
>> I don't find teaching children is a laughing matter, neither is the idea
>> of children learning to code.
>> In Canada, we have initiatives like Girls Learning Code and Kids
>> Learning Code. I mentored in a couple of those events and the students
>> are girls aged 8-14. They surprised me with their abilities to learn. I
>> would suggest looking for such mentoring opportunities in your area to
>> gain appreciation with this regard.
>> Thanks.
>> (Sorry to derail everyone from the topic of list comprehension. Please
>> continue!)
>>
>>
> The RUE was allowed to insult the community for years and got away with
> it.  I'm autistic, stepped across the line, and got hammered.  Hypocrisy at
> its best.  Even funnier, the BDFL has asked for my advice in recent weeks
> with respect to the bug tracker.  I've replied, giving the bare minimum
> that I feel I can give within the circumstances.
>
> Yours most disgustingly.
>
> Mark Lawrence.
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Mark Lawrence via Python-ideas]

On 16/10/2016 16:41, Mariatta Wijaya wrote:

Her reaction was hilarious:

"Whom does he teach? Children?"


I sense mockery in your email, and it does not conform to the PSF code
of conduct. Please read the CoC before posting in this mailing list. The
link is available at the bottom of every python mailing list
email.https://www.python.org/psf/codeofconduct/

I don't find teaching children is a laughing matter, neither is the idea
of children learning to code.
In Canada, we have initiatives like Girls Learning Code and Kids
Learning Code. I mentored in a couple of those events and the students
are girls aged 8-14. They surprised me with their abilities to learn. I
would suggest looking for such mentoring opportunities in your area to
gain appreciation with this regard.
Thanks.
(Sorry to derail everyone from the topic of list comprehension. Please
continue!)



The RUE was allowed to insult the community for years and got away with 
it.  I'm autistic, stepped across the line, and got hammered.  Hypocrisy 
at its best.  Even funnier, the BDFL has asked for my advice in recent 
weeks with respect to the bug tracker.  I've replied, giving the bare 
minimum that I feel I can give within the circumstances.


Yours most disgustingly.

Mark Lawrence.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Mariatta Wijaya]

>Her reaction was hilarious:
>
>"Whom does he teach? Children?"

I sense mockery in your email, and it does not conform to the PSF code of
conduct. Please read the CoC before posting in this mailing list. The link
is available at the bottom of every python mailing list email.
https://www.python.org/psf/codeofconduct/
I don't find teaching children is a laughing matter, neither is the idea of
children learning to code.
In Canada, we have initiatives like Girls Learning Code and Kids Learning
Code. I mentored in a couple of those events and the students are girls
aged 8-14. They surprised me with their abilities to learn. I would suggest
looking for such mentoring opportunities in your area to gain appreciation
with this regard.
Thanks.
(Sorry to derail everyone from the topic of list comprehension. Please
continue!)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[David Mertz]

On Sun, Oct 16, 2016 at 5:34 AM, Sven R. Kunze  wrote:

> On 16.10.2016 07:08, David Mertz wrote:
>
>> In case it wasn't entirely clear, I strongly and vehemently opposed this
>> unnecessary new syntax. It is confusing, bug prone, and would be difficult
>> to teach.
>
>

> "Whom does he teach? Children?"
> Me: "What? No, everybody I think. Why?"
> She: "It's easy enough to remember what the star does."
>

As I've said, the folks I teach are mostly working scientists with
doctorates in scientific fields and years of programming experience in
languages other than Python.  For example, rocket scientists at NASA.

Now I admit that I don't specifically know how quickly they would pick up
on something I've never taught them.  But I've written enough teaching
materials and articles and books and I have a certain intuition.  The way
you explained the special case you built up is pretty good, but it's very
tailored to making that specific case plausible, and is not general.


> She also asked what would the alternative would look like. I wrote:
> """
> >>> from itertools import chain
> >>> list(chain.from_iterable((i,i,i) for i in range(4)))
> [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3]
> """
> Her reaction was like: "That's supposed to be easy to remember? I find the
> star easier."
>

That is an absolutely terrible construct, obviously.  I have to pause and
think a while myself to understand what it does.

It also answers a very different use case than the one that has been mostly
discussed in this thread.  A much better spelling is:

>>> [i for i in range(4) for _ in range(3)]
[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3]

For some related uses, itertools.repeat() is very useful.

But the case that has been discussed far more often is like this:

>>> listOfLists = [[1,2,3], [4,5,6,7], [8,9]]
>>> flatten(listOfLists)

The very long argument is that somehow that would be easier to spell as:

>>> [*i for i in listOfLists]

It's just not easier.  And there are contrary intuitions that will occur to
many people based on the several other related-but-different uses of * for
packing/unpacking in other contexts.

Also, this simplest case might be teachable, but the more general "exactly
where can I use that star in a comprehensions" will be far harder to
explain plausibly.  What's the pattern here?

>>> [(*i,) for i in listOfLists]
[(1, 2, 3), (4, 5, 6, 7), (8, 9)]
>>> [(i,*i) for i in listOfLists]
[([1, 2, 3], 1, 2, 3), ([4, 5, 6, 7], 4, 5, 6, 7), ([8, 9], 8, 9)]
>>> [(*i,*i) for i in listOfLists]
[(1, 2, 3, 1, 2, 3), (4, 5, 6, 7, 4, 5, 6, 7), (8, 9, 8, 9)]
>>> [*(i,) for i in listOfLists]
# ... no clear intuition here ...
>>> [*(*i) for i in listOfLists
# ... even more confusing ...

Yes! I know those first few are actually doing something different than the
proposed new syntax.

But explaining that to my rocket scientists or your girlfriend in a
consistent and accurate way would be a huge challenge.  It would mostly
come down to "don't do that, it's too confusing."

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Ivan Levkivskyi]

On 16 October 2016 at 06:47, Chris Angelico  wrote:

> On Sun, Oct 16, 2016 at 3:44 PM, Greg Ewing 
> wrote:
> > Steven D'Aprano wrote:
> >
> >> This thread is a huge, multi-day proof that people do not agree that
> this
> >> is a "reasonable" interpretation.
> >
> >
> > So far I've seen one very vocal person who disgrees, and
> > maybe one other who isn't sure.
> >
>
> And what you're NOT seeing is a whole lot of people (myself included)
> who have mostly glazed over, unsure what is and isn't reasonable, and
> not clear enough on either side of the debate to weigh in. (Or not
> even clear what the two sides are.)
>
>
+1

There are lots of arguments of whether the new syntax is readable or not
etc.,
but not so many arguments why we need this and what kind of problems it
would solve.

What I have learned from this megathread is that the syntax [*foo for foo
in bar]
is proposed as a replacement for a one-liner itertools.chain(*[foo for foo
in bar]).
I do not have any strong opinion on this, because I simply do not use
such constructs frequently (if ever).

--
Ivan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Sven R. Kunze]

On 16.10.2016 07:08, David Mertz wrote:


In case it wasn't entirely clear, I strongly and vehemently opposed 
this unnecessary new syntax. It is confusing, bug prone, and would be 
difficult to teach.




As this discussion won't come to an end, I decided to consult my girlfriend.

I started with (btw. she learned basic Python to solve some math quizzes):
"""
Let's install a list in another one.

>>> meine_liste
[2, 3, 4, 5]
>>> ['a', meine_liste, 'b']
['a', [2, 3, 4, 5], 'b']

Maybe, we want to remove the brackets.

>>> ['a', *meine_liste, 'b']
['a', 2, 3, 4, 5, 'b']

Now, the problem of the discussion is the following:

>>> [(i,i,i) for i in range(4)]
[(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3)]

Let's remove these inner parentheses again.

>>> [*(i,i,i) for i in range(4)]
  File "", line 1
SyntaxError: iterable unpacking cannot be used in comprehension

Some guy wanted to remove that restriction.
"""

I said a teacher contributed to the discussion and he finds this too 
complicated and confusing and does not even teach * in list displays at 
all. Her reaction was hilarious:


"Whom does he teach? Children?"
Me: "What? No, everybody I think. Why?"
She: "It's easy enough to remember what the star does."


She also asked what would the alternative would look like. I wrote:
"""
>>> from itertools import chain
>>> list(chain.from_iterable((i,i,i) for i in range(4)))
[0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3]
"""
Her reaction was like: "That's supposed to be easy to remember? I find 
the star easier."


In the end, she also added: "Not everybody drives a car but they still 
exist."


Cheers,
Sven

PS: off to the weekend. She's already complaint that I should spend less 
time inside my mailbox and more with her. ;)

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Steven D'Aprano]

On Sun, Oct 16, 2016 at 05:14:54AM +, Neil Girdhar wrote:

[Steven (me), refering to Greg]
> > Because as your own email inadvertently reinforces, if sequence
> > unpacking made sense in the context of a list comprehension, it would
> > already be allowed rather than a SyntaxError: it is intentionally
> > prohibited because it doesn't make sense in the context of list comps.
>
> 
> Whoa, hang on a second there.  It is intentionally prohibited because
> Joshua Landau (who helped a lot with writing and implementing the PEP) and
> I felt like there was going to be a long debate and we wanted to get PEP
> 448 checked in.
> 
> If it "didn't make sense" as you say, then we would have said so in the
> PEP. Instead, Josh wrote:
> 
> This was met with a mix of strong concerns about readability and mild
> support. In order not to disadvantage the less controversial aspects of the
> PEP, this was not accepted with the rest of the proposal.

Okay, interesting, and thanks for the correction.

> I don't remember who it was who had those strong concerns (maybe you?)  But
> that's why we didn't include it.

I'm pretty sure it wasn't me. I don't recall being involved at all with 
any discussions about PEP 448, and a quick search has failed to come up 
with anything relevant. I think I sat that one out.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Neil Girdhar]

On Sat, Oct 15, 2016 at 9:42 PM Steven D'Aprano  wrote:

> On Sun, Oct 16, 2016 at 12:48:36PM +1300, Greg Ewing wrote:
> > Steven D'Aprano wrote:
> > >Are you now supporting my argument that starring the list comprehension
> > >expression isn't meaningful?
> >
> > The context it's in (a form of list display) has a clear
> > meaning for a comma-separated list of values, so there
> > is a reasonable interpretation that it *could* be given.
>
> This thread is a huge, multi-day proof that people do not agree that
> this is a "reasonable" interpretation.
>
>
> > >py> iterable = [(1, 'a'), (2, 'b')]
> > >py> [(100, *t) for t in iterable]
> > >[(100, 1, 'a'), (100, 2, 'b')]
> >
> > The * there is in the context of constructing a tuple,
> > not the list into which the tuple is placed.
>
> Right: the context of the star is meaningful. We all agree that *t in a
> list display [a, b, c, ...] is meaningful; same for tuples; same for
> function calls; same for sequence unpacking for assignment.
>
> What is not meaningful (except as a Perlish line-noise special case to
> be memorised) is *t as the list comprehension expression.
>
> I've never disputed that we could *assert* that *t in a list comp means
> "flatten". We could assert that it means anything we like. But it
> doesn't follow from the usual meaning of sequence unpacking anywhere
> else -- that's why it is currently a SyntaxError, and that's why people
> reacted with surprise at the OP who assumed that *t would magically
> flatten his iterable. Why would you assume that? It makes no sense to me
> -- that's not how sequence unpacking works in any other context, it
> isn't how list comprehensions work.
>
> Right from the beginning I called this "wishful thinking", and *nothing*
> since then has changed my mind. This proposal only makes even a little
> bit of sense if you imagine list comprehensions
>
> [*t for a in it1 for b in it2 for c in it3 ... for t in itN]
>
> completely unrolled into a list display:
>
> [*t, *t, *t, *t, ... ]
>
> but who does that? Why would you reason about your list comps like that?
> If you think about list comps as we're expected to think of them -- as
> list builders equivalent to a for-loop -- the use of *t there is
> invalid. Hence it is a SyntaxError.
>
> You want a second way to flatten your iterables? A cryptic, mysterious,
> Perlish line-noise way? Okay, fine, but don't pretend it is sequence
> unpacking -- in the context of a list comprehension, sequence unpacking
> doesn't make sense, it is invalid. Call it something else: the new
> "flatten" operator:
>
> [^t for t in iterable]
>
> for example, which magically adds an second invisible for-loop to your
> list comps:
>
> # expands to
> for t in iterable:
> for x in t:
> result.append(x)
>
> Because as your own email inadvertently reinforces, if sequence
> unpacking made sense in the context of a list comprehension, it would
> already be allowed rather than a SyntaxError: it is intentionally
> prohibited because it doesn't make sense in the context of list comps.
>

Whoa, hang on a second there.  It is intentionally prohibited because
Joshua Landau (who helped a lot with writing and implementing the PEP) and
I felt like there was going to be a long debate and we wanted to get PEP
448 checked in.

If it "didn't make sense" as you say, then we would have said so in the
PEP. Instead, Josh wrote:

This was met with a mix of strong concerns about readability and mild
support. In order not to disadvantage the less controversial aspects of the
PEP, this was not accepted with the rest of the proposal.

I don't remember who it was who had those strong concerns (maybe you?)  But
that's why we didn't include it.

Best,

Neil


>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/ROYNN7a5VAc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Chris Angelico]

On Sun, Oct 16, 2016 at 3:44 PM, Greg Ewing  wrote:
> Steven D'Aprano wrote:
>
>> This thread is a huge, multi-day proof that people do not agree that this
>> is a "reasonable" interpretation.
>
>
> So far I've seen one very vocal person who disgrees, and
> maybe one other who isn't sure.
>

And what you're NOT seeing is a whole lot of people (myself included)
who have mostly glazed over, unsure what is and isn't reasonable, and
not clear enough on either side of the debate to weigh in. (Or not
even clear what the two sides are.)

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[David Mertz]

On Oct 15, 2016 6:42 PM, "Steven D'Aprano"  wrote:
> doesn't make sense, it is invalid. Call it something else: the new
> "flatten" operator:
>
> [^t for t in iterable]
>
> for example, which magically adds an second invisible for-loop to your
list comps:

This thread is a lot of work to try to save 8 characters in the spelling of
`flatten(it)`. Let's just use the obvious and intuitive spelling.

We really don't need to be Perl. Folks who want to write Perl have a
perfectly good interpreter available already.

The recipes in itertools give a nice implementation:

def flatten(listOfLists):
"Flatten one level of nesting"
return chain.from_iterable(listOfLists)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Steven D'Aprano]

On Sun, Oct 16, 2016 at 12:48:36PM +1300, Greg Ewing wrote:
> Steven D'Aprano wrote:
> >Are you now supporting my argument that starring the list comprehension 
> >expression isn't meaningful?
> 
> The context it's in (a form of list display) has a clear
> meaning for a comma-separated list of values, so there
> is a reasonable interpretation that it *could* be given.

This thread is a huge, multi-day proof that people do not agree that 
this is a "reasonable" interpretation.


> >py> iterable = [(1, 'a'), (2, 'b')]
> >py> [(100, *t) for t in iterable]
> >[(100, 1, 'a'), (100, 2, 'b')]
> 
> The * there is in the context of constructing a tuple,
> not the list into which the tuple is placed.

Right: the context of the star is meaningful. We all agree that *t in a 
list display [a, b, c, ...] is meaningful; same for tuples; same for 
function calls; same for sequence unpacking for assignment.

What is not meaningful (except as a Perlish line-noise special case to 
be memorised) is *t as the list comprehension expression.

I've never disputed that we could *assert* that *t in a list comp means 
"flatten". We could assert that it means anything we like. But it 
doesn't follow from the usual meaning of sequence unpacking anywhere 
else -- that's why it is currently a SyntaxError, and that's why people 
reacted with surprise at the OP who assumed that *t would magically 
flatten his iterable. Why would you assume that? It makes no sense to me 
-- that's not how sequence unpacking works in any other context, it 
isn't how list comprehensions work.

Right from the beginning I called this "wishful thinking", and *nothing* 
since then has changed my mind. This proposal only makes even a little 
bit of sense if you imagine list comprehensions

[*t for a in it1 for b in it2 for c in it3 ... for t in itN]

completely unrolled into a list display:

[*t, *t, *t, *t, ... ]

but who does that? Why would you reason about your list comps like that? 
If you think about list comps as we're expected to think of them -- as 
list builders equivalent to a for-loop -- the use of *t there is 
invalid. Hence it is a SyntaxError.

You want a second way to flatten your iterables? A cryptic, mysterious, 
Perlish line-noise way? Okay, fine, but don't pretend it is sequence 
unpacking -- in the context of a list comprehension, sequence unpacking 
doesn't make sense, it is invalid. Call it something else: the new 
"flatten" operator:

[^t for t in iterable]

for example, which magically adds an second invisible for-loop to your 
list comps:

# expands to
for t in iterable:
for x in t:
result.append(x)

Because as your own email inadvertently reinforces, if sequence 
unpacking made sense in the context of a list comprehension, it would 
already be allowed rather than a SyntaxError: it is intentionally 
prohibited because it doesn't make sense in the context of list comps.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Martti Kühne]

On Sat, Oct 15, 2016 at 12:48 PM, Steven D'Aprano  wrote:
> Oh look, just like now:
>
> py> iterable = [(1, 'a'), (2, 'b')]
> py> [(100, *t) for t in iterable]
> [(100, 1, 'a'), (100, 2, 'b')]
>
> Hands up anyone who expected to flatten the iterable and get
>
> [100, 1, 'a', 100, 2, 'b']
>
> instead? Anyone? No?
>

I don't know whether that should be provocating or beside the poinnt.
It's probably both. You're putting two expectations on the same
example: first, you make the reasonable expectation that results in
[(100, 1, 'a'), (100, 2, 'b')], and then you ask whether anyone
expected [100, 1, 'a', 100, 2, 'b'], but don't add or remove anything
from the same example. Did you forget to put a second example using
the new notation in there?
Then you'd have to spell it out and start out with [*(100, *t) for t
in iterable]. And then you can ask who expected [100, 1, 'a', 100, 2,
'b']. Which is what this thread is all about.

cheers!
mar77i
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Greg Ewing]

Martti Kühne wrote:

You brush over the fact that *t is not limited to a replacement by a
comma-separated sequence of items from t, but *t is actually a
replacement by that comma-separated sequence of items from t INTO an
external context.


Indeed. In situations where there isn't any context for
the interpretation of *, it's not allowed. For example:

>>> x = *(1, 2, 3)
  File "", line 1
SyntaxError: can't use starred expression here

But

>>> x = 1, *(2, 3)
>>> x
(1, 2, 3)

The * is allowed there because it's already in a context
where a comma-separated list has meaning.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Martti Kühne]

On Sat, Oct 15, 2016 at 10:09 AM, Steven D'Aprano  wrote:
> Not everything is a function. What's your point?
>
> As far as I can see, in *every* other use of sequence unpacking, *t is
> conceptually replaced by a comma-separated sequence of items from t. If
> the starred item is on the left-hand side of the = sign, we might call
> it "sequence packing" rather than unpacking, and it operates to collect
> unused items, just like *args does in function parameter lists.
>


You brush over the fact that *t is not limited to a replacement by a
comma-separated sequence of items from t, but *t is actually a
replacement by that comma-separated sequence of items from t INTO an
external context. For func(*t) to work, all the elements of t are kind
of "leaked externally" into the function argument list's context, and
for {**{'a': 1, 'b': 2, ...}} the inner dictionary's items are kind of
"leaked externally" into the outer's context.

You can think of the */** operators as a promotion from append to
extend, but another way to see this is as a promotion from yield to
yield from. So if you want to instead of append items to a
comprehension, as is done with [yield_me for yield_me in iterator],
you can see this new piece as a means to [*yield_from_me for
yield_from_me in iterator]. Therefore I think it's a bit confusing
that yield needs a different keyword if these asterisk operators
already have this intuitive promotion effect.

Besides, [*thing for thing in iterable_of_iters if cond] has this cool
potential for the existing any() and all() builtins for cond, where a
decision can be made based on the composition of the in itself
iterable thing.

cheers!
mar77i
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/