Re: [Python-ideas] Shuffled

[Danilo J. S. Bellini]

>
> 1. The cognitive leap between shuffling and sampling isn't small
>
I don't think so, actually the Fisher-Yates shuffle algorithm algorithm is
an iterative sampling algorithm: https://gist.github.com/danilo
bellini/6384872


> 2. "shuffled" would be a more logical name for an out-of-place shuffle
> than "sample"
>
Agreed.


> 3. "Copy the whole container" would be a surprising default for a
> function called "sample"
>
Perhaps "sample the whole population" sounds strange. Indeed, statisticians
probably wouldn't be happy with that. That reminds me the variance
estimator with the extra unbiasing "-1" in the denominator versus the
population variance.


> 4. With a default, random.sample becomes more easily confused with
> random.choice
>
I don't think k=1 would be a good default sample size from a statistics
point of view, but I get the point (I'm from a DSP background, where "a
sample" means one single "value").

Controling the random function is required for the function to be really
pure, else its output won't depend only on the inputs (and there would be
some "state" in that "implicit input"). That would also be a great feature
when non-uniform (or external) random number generators are to be used.
This seem to be something that only shuffle gives some control (among the
functions we're talking about), or am I missing something?


2016-09-08 1:26 GMT-03:00 Nick Coghlan :

> On 8 September 2016 at 13:33, Danilo J. S. Bellini
>  wrote:
> > Nice to know about random.sample! =)
> >
> > I think what OP said can then be reduced to having the default k in
> > random.sample to be the iterable size. The existance of random.sample is
> a
> > very strong argument against "shuffled", and the only "feature" shuffled
> > would have that random.sample doesn't have is that default size.
>
> There are a few reasons I don't think defining a default for
> random.sample() would be a good answer to Arek's question:
>
> 1. The cognitive leap between shuffling and sampling isn't small
> 2. "shuffled" would be a more logical name for an out-of-place shuffle
> than "sample"
> 3. "Copy the whole container" would be a surprising default for a
> function called "sample"
> 4. With a default, random.sample becomes more easily confused with
> random.choice
>
> For the last two concerns, if I saw "result =
> random.sample(container)" and didn't already know about
> random.choice(), I'd expect it to behave like random.choice(). Even
> knowing they're different, I'd still need to do a double-take to make
> sure I was remembering which was which correctly. By contrast,
> "random.sample(container, 1)", "random.sample(container, k)",
> "random.sample(container, len(container))" are all clearly different
> from the single result primitive "random.choice(container)"
>
> One interesting (to me anyway) aspect of an out-of-place shuffle is
> that you can readily implement it based on *either* of the more
> primitive operations (in-place shuffle or random sample):
>
> def shuffled(container):
> result = list(container)
> random.shuffle(result)
> return result
>
> def shuffled(container):
> return random.sample(container, len(container))
>
> Writing down those two examples does highlight a potential refactoring
> benefit to having "out-of-place shuffle" as an explicitly represented
> concept in the core Random API: it can be defined in terms of shuffle
> or sample *on the same Random instance*, and hence automatically
> benefit when code switches from using the global random number
> generator to using a purpose-specific Random instance that allows
> greater control over the reproducibility of results (which can be very
> important for testing, games, modeling & simulation).
>
> The above helper functions are both flawed on that front: they
> hardcode the use of the default global random number generator. To
> permit use of a specific random instance, they need to be changed to:
>
> def shuffled(container, random=random):
> result = list(container)
> random.shuffle(result)
> return result
>
> def shuffled(container, random=random):
> return random.sample(container, len(container))
>
> and then used as "result = shuffled(original, my_random_instance)" if
> you decide to switch away from the global API.
>
> By contrast, a method based implementation could be refactored the
> exact same way as any other random.Random method:
>
> result = my_random_instance.shuffled(original)
>
> I'll reiterate that I don't have a use case for this myself, but I'll
> cite the key arguments I see in favour:
>
> - an out-of-place shuffle may be useful as a bridging concept between
> in place shuffling and out of place sampling (in either direction)
> - the presence of "shuffled" becomes a reminder that "shuffle" itself
> is an in-place operation
> - "write it yourself" isn't as simple as it first sounds due to the
> common migration away 

Re: [Python-ideas] Shuffled

[Nick Coghlan]

On 8 September 2016 at 13:33, Danilo J. S. Bellini
 wrote:
> Nice to know about random.sample! =)
>
> I think what OP said can then be reduced to having the default k in
> random.sample to be the iterable size. The existance of random.sample is a
> very strong argument against "shuffled", and the only "feature" shuffled
> would have that random.sample doesn't have is that default size.

There are a few reasons I don't think defining a default for
random.sample() would be a good answer to Arek's question:

1. The cognitive leap between shuffling and sampling isn't small
2. "shuffled" would be a more logical name for an out-of-place shuffle
than "sample"
3. "Copy the whole container" would be a surprising default for a
function called "sample"
4. With a default, random.sample becomes more easily confused with random.choice

For the last two concerns, if I saw "result =
random.sample(container)" and didn't already know about
random.choice(), I'd expect it to behave like random.choice(). Even
knowing they're different, I'd still need to do a double-take to make
sure I was remembering which was which correctly. By contrast,
"random.sample(container, 1)", "random.sample(container, k)",
"random.sample(container, len(container))" are all clearly different
from the single result primitive "random.choice(container)"

One interesting (to me anyway) aspect of an out-of-place shuffle is
that you can readily implement it based on *either* of the more
primitive operations (in-place shuffle or random sample):

def shuffled(container):
result = list(container)
random.shuffle(result)
return result

def shuffled(container):
return random.sample(container, len(container))

Writing down those two examples does highlight a potential refactoring
benefit to having "out-of-place shuffle" as an explicitly represented
concept in the core Random API: it can be defined in terms of shuffle
or sample *on the same Random instance*, and hence automatically
benefit when code switches from using the global random number
generator to using a purpose-specific Random instance that allows
greater control over the reproducibility of results (which can be very
important for testing, games, modeling & simulation).

The above helper functions are both flawed on that front: they
hardcode the use of the default global random number generator. To
permit use of a specific random instance, they need to be changed to:

def shuffled(container, random=random):
result = list(container)
random.shuffle(result)
return result

def shuffled(container, random=random):
return random.sample(container, len(container))

and then used as "result = shuffled(original, my_random_instance)" if
you decide to switch away from the global API.

By contrast, a method based implementation could be refactored the
exact same way as any other random.Random method:

result = my_random_instance.shuffled(original)

I'll reiterate that I don't have a use case for this myself, but I'll
cite the key arguments I see in favour:

- an out-of-place shuffle may be useful as a bridging concept between
in place shuffling and out of place sampling (in either direction)
- the presence of "shuffled" becomes a reminder that "shuffle" itself
is an in-place operation
- "write it yourself" isn't as simple as it first sounds due to the
common migration away from the random module functions to a custom
Random instance

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Steven D'Aprano]

On Wed, Sep 07, 2016 at 11:47:49PM -0300, Danilo J. S. Bellini wrote:
> Though I agree with the argument that inexperienced developers are
> [usually] worse, that's not the case here, unless anyone here is really
> trying to say the ones arguing for or against "shuffled" are inexperienced.
> These ad hominem won't bring us anywhere.

That is not what I said. Please be careful of accusing others of things 
they didn't say.

It was Sven who made the claim that inexperienced developers are better 
at seeing inconsistancies. I was responding specifically to that claim. 
I'm not saying anything about the experience, or lack of, of either Sven 
or Arek.


> No one seem to be arguing if "shuffled" is inconsistent, but the "shuffle"
> consistency is a subject that didn't reach consensus. And the point now
> seem to be whether "shuffled" is useful or not. If I understood correctly,
> that proposal is about writing code in a more functional style (or a
> expression-oriented programming style), with a pure function for shuffling.
> 
> I think there are other related subjects that can be said about
> sorted/shuffled. For example: why a list? Why not a generator? 

You cannot sort a sequence lazily. You have to sort the entire sequence 
before you can be sure which item comes first. Making sorted() an 
iterator by default doesn't give you any advantage: it still has to 
sort the entire sequence before it can yield the first item.


> Something
> like random.choice is useful to get one single value, but if one just want
> to see a few random values from a large input without repeating the values,
> what's the best approach?

random.sample()

 
> Does that mean you expect Python to provide appended() [...] extended()
> > [...]
> >
> Doesn't the "+" operator do that?

Of course it does. But the "consistency" argument would be "why is 
append a method not an operator? Why isn't + called appended?".

That's just spelling. It isn't that important.


> > [...] list.cleared()?
> 
> Huh? OP is clearly talking about avoiding state, avoiding "in-place"
> operations. A clear list would be an empty list [].

Of course. We can see that. But again, that's "inconsistent":

* the in-place operation is spelled `list.clear()`

* the functional operation is spelled `[]`

There is nothing wrong with that. The value of consistency is grossly 
exaggerated. Sometimes it really doesn't matter if similar operations 
are spelled differently.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Nick Coghlan]

On 8 September 2016 at 12:00, Steven D'Aprano  wrote:
> On Wed, Sep 07, 2016 at 11:43:59PM +0200, Sven R. Kunze wrote:
>
>> BUT experienced devs also need to recognize and respect the fact that
>> younger/unexperienced developers are just better in detecting
>> inconsistencies and bloody work-arounds.
>
> That is not a fact. It is the opposite of a fact -- inexperienced
> developers are WORSE at spotting inconsistencies, because they don't
> recognise deep consistencies.

There are also cases where we're internally inconsistent or lacking
internal integration because we're being consistent with different
external environments.

So we have both binary floating point and decimal floating point
because there are inherent conflicts in the underlying computational
models.

We have both the abc module (which deals strictly with runtime
classes) and the typing module (which deals strictly with type
categories) as those things are similar, but not the same.

We have both asyncio and traditional synchronous APIs as there are
essential conflicts between blocking and non-blocking models of
control flow.

Cross-platform secure network connectivity handling is a pain because
many of the underlying platform APIs are unreliably implemented and
poorly documented moving targets.

Cross-platform software distribution is a pain largely because major
platform vendors are typically competing to lock in developers, so
their incentives are aligned with making what we're trying to do
difficult rather than offering to help out.

For these kinds of cases, frustrations on the part of experienced
developers arise when folks ask entirely reasonable questions "Why is
 so complicated?", but then refuse to accept the answer of
"Because it's genuinely complex". Pretending complexity doesn't exist
doesn't help anyone - it means they're left to their own devices to
come up with answers, rather than receiving more explicit guidance at
the language level.

More directly relevant to the current discussion is the principle of
"Not every three line function needs to be a builtin (or even in the
standard library)". There are ways to collect objective data on the
relative usage of particular constructs (especially as hosting
platforms like GitHub team up with analytics platforms like Google Big
Query to make their data searchable, and initiatives like Software
Heritage aim to bring that kind of capability to all published open
source software), but one of the *simplest* criteria to apply, and the
one that experienced core developers can typically answer without
further research, is "Would the standard library benefit from this?".

When the answer to that is "No", then the default answer to requests
for API additions that are relatively easy for people to implement for
themselves is always going to be "No" - otherwise the stdlib will
collapse under its own weight (and the persistent "up and to the
right" trend on http://bugs.python.org/issue?@template=stats means
that there's a solid argument to be made that the standard library is
already bigger than we can effectively maintain at current maintenance
funding levels).

Cheers,
Nick.

P.S. I wrote 
http://www.curiousefficiency.org/posts/2011/04/musings-on-culture-of-python-dev.html
about this a few years ago, and I still think it's an accurate
description of the underlying causes of these conflicts

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Steven D'Aprano]

On Thu, Sep 08, 2016 at 12:22:28AM +0200, Sven R. Kunze wrote:

> About the confusion of returning None. It's not confusing in the sense 
> of "o my god, I did it wrong, I need to learn it", but more, like "I 
> used shuffle, BECAUSE it's the only one I could find in a hurry"

Then don't be in such a hurry.

Honestly, there is no substitute for professionalism. Outside of fantasy 
novels and movies, there has never been a "ticking time bomb" situation 
where a programmer has to write code to shuffle a list in six seconds or 
the world will be destroyed. There is *always* time to at least glance 
at the docs, and it is never more than a handful of keypresses away in 
the interactive interpreter:

help(random.shuffle)

And if you don't read the docs? Oh well, you find out soon enough once 
you actually test your code or have a code review. No harm done.

If the programmer doesn't do *any* of those things -- doesn't read the 
docs, doesn't test their code, doesn't do code reviews -- then there's 
no hope for them. We could provide shuffled() and shuffle() and they'll 
still end up picking the wrong one fifty percent of the time.

"My wild guess of how a function works was wrong, so now I'm cheesed off 
that the function doesn't do what I expect" is NOT a good reason to add 
things to the standard library.

> I would expect this from 
> Python as it already provides both alternatives for sorting".

Does that mean you expect Python to provide appended() as well as 
append(), extended() as well as extend(), popped() as well as pop(), 
cleared() as well as clear(), inserted() as well as insert(), removed() 
as well as remove()?

If your response is to say "Why do we need all those? Why on earth would 
anyone need to use list.cleared()?" then perhaps you can now understand 
why shuffled() has to prove itself too. Functions don't get added just 
because they *can* be added. They have to be worth it. Every function 
has costs as well as benefits. Each function adds:

- more code to download, compile, maintain
- more tests
- more documentation
- more for the user to learn
- more choices for the user to decide between
- more complexity
- more risk of bugs

Unless the proposal can demonstrate that the benefit outweighs the 
costs, it should not be added. Merely stating that you need it 
doesn't demonstate that a four-line helper function at the top of 
your module isn't sufficient.

For what's it is worth, if it were *my* call, I'd accept that the costs 
of adding this are low, but the benefits are just a *tiny* bit higher. 
But that's a judgement call, and if Raymond see the cost:benefit ratio 
going the other way, I have no objective argument to change his mind.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Typecheckers: there can be only one

[Nick Coghlan]

On 8 September 2016 at 07:31, Hugh Fisher  wrote:
> On Wed, Sep 7, 2016 at 9:56 PM, Nick Coghlan  wrote:
>>
>> Exactly - in a dynamic language like Python, running a typechecker is
>> a form of testing, rather than a necessary part of getting the code to
>> work in the first place.
>>
>
> The average programmer such as myself will expect that if I write code
> specifying the type of a variable or whatever it should *do
> something*. It's code, I wrote it, it should be interpreted.
>
> Assuming PEP 526 is implemented, suppose as a very new programmer I write
>
> foo: dict = 0
>
> (If anyone thinks even new programmers wouldn't write that, you've
> never taught an introductory programming course.)
>
> It takes very little programming experience to know that is flat out
> wrong. I cannot think of any other programming language with type
> notation where this would not be immediately flagged as an error. But
> Python will silently accept it?

It wouldn't surprise me in the least if education focused environments
like the Mu text editor and potentially even Jupyter Notebooks decide
to start running a typechecker implicitly, as many IDEs already do
that. That way, if students do add type annotations (even if they're
missing from all of the example code presented), they'll be enforced
automatically.

That's a decision to be made by instructors and development
environment developers based on their assessment of their target
audience though, it isn't something that needs to be baked into the
core runtime design.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Steven D'Aprano]

On Wed, Sep 07, 2016 at 11:43:59PM +0200, Sven R. Kunze wrote:

> BUT experienced devs also need to recognize and respect the fact that 
> younger/unexperienced developers are just better in detecting 
> inconsistencies and bloody work-arounds.

That is not a fact. It is the opposite of a fact -- inexperienced 
developers are WORSE at spotting inconsistencies, because they don't 
recognise deep consistencies.

For example, the on-going controversy about mutable default arguments:

def func(x=0, y=[]):
x += 1
y.append(1)
print(x, y)

To the inexperienced Python developer this makes no sense. Why does 
Pythod reset the value of x by default, while y remembers the value it 
had from the previous call? That's an inconsistency.

But the more experienced dev realises that there is no inconsistency. 
Python behaves exactly the same way in both cases. Both x and y are 
treated the same: they both are treated as "Early Binding". The only 
difference is that 0 is immutable, and so x += 1 binds a new value to x, 
while [] is mutable, and y.append(1) modifies that value in place.

Try this instead:

def func(x=0, y=[]):
x += 1
y = y + [1]  # don't use +=
print(x, y)

The more experienced programmer might not *like* this behaviour, they 
may wish that Python used Late Binding instead, but it takes a very 
subtle and deep knowledge of the language to understand that this is not 
an inconsistency despite the superficial appearances.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Alexander Belopolsky]

On Wed, Sep 7, 2016 at 9:12 PM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

> On Wed, Sep 7, 2016 at 8:14 PM, Arek Bulski  wrote:
> >
> > If you want to see the declarative tests, here it is.
> > https://github.com/construct/construct/blob/master/tests/test_all.py
>
>
> So, why can't you call random.shuffle(all_tests) if you want to run your
> tests in random order?


It may be instructive for you to see how this functionality is implemented
in CPython's own test suit:

https://github.com/python/cpython/blob/276f4ef97a434d4279a2d207daa34cafcf0994f7/Lib/test/libregrtest/main.py#L230
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Alexander Belopolsky]

On Wed, Sep 7, 2016 at 6:56 PM, Arek Bulski  wrote:

> In the project I maintain (construct) there are declarative testcases that
> look like a long list of (func, args, excepted output, error type) tuples.
> There is no way for me to call shuffle in there.


Can you explain why?  Something like this can be easily done with pytest:

In [1]: def foo(x):
   ...: return x + 1
   ...:

In [2]: import pytest

In [3]: @pytest.mark.parametrize('x, y', [
   ...: (100, 101),
   ...: (200, 201),
   ...: (300, 301),])
   ...: def test_foo(x, y):
   ...: assert foo(x) == y
   ...:

In [4]: test_foo.parametrize.args
Out[4]: ('x, y', [(100, 101), (200, 201), (300, 301)])

In [5]: import random

In [6]: random.shuffle(test_foo.parametrize.args[1])

In [7]: test_foo.parametrize.args
Out[7]: ('x, y', [(200, 201), (100, 101), (300, 301)])
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[David Mertz]

On Wed, Sep 7, 2016 at 3:10 PM, Sven R. Kunze  wrote:

> @David
> Your idea of a PyPI package could almost work. However, in an interactive
> python console, I expect as much batteries included as possible to make it
> as quick as possible. And considering how simple such wrapper would be, it
> almost does not warrant the download of a third-party package.
>

Of course I don't think it's worth having a PyPI package for this one
function.  That's why I suggested it could possibly live in `boltons` (that
already contains a lot of other simple functions that aren't in the stdlib)
or in `arek_utils` (which I haven't described except implicitly as a
collection of "a bunch of useful things").

Tim argues that this functionality is even too simple for inclusion in
boltons; that might be true.  But either way, it's not a huge burden in the
interactive console to type a first line of `from my_utils import *` before
getting down to work.



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Arek Bulski]

Thank you all for the meta discussion. I know we sidetracked a bit but
please everyone remember, this is an open discussion floor.

I can present you a use case, just didnt think I even needed one. This
seems so obvious to me that it should just go in without much saying. In
the project I maintain (construct) there are declarative testcases that
look like a long list of (func, args, excepted output, error type) tuples.
There is no way for me to call shuffle in there. I can only use shuffled.
So here it is, the use case you requested.

The argument that it can be easily implemented does not stick. In
interactive sessions, indeed writing own functions is a pain in the ass.
Expecting batteries included is rightful. Builtins like min() max() sum()
are sooo easy to implement on your own. So are some methods in itertools,
you can even find their sources in the docs. That doesnt mean batteries
should not be included.

shuffled = sorted(list, key=lambda x: randint)
This is wrong because sorting is comparison based and must be nlogn bound.
Shuffling is cheaper computationally. Also someone would have to come up
with a formal proof that it provides a uniform distribution which probably
was done for the shuffle implementaion.

Another argument still remains: there is a missing analogy between
sort-sorted and shuffle-shuffled.

I did NOT suggest making it a builtin. It should be added to random module.


>  "Also shuffle() should return self so mutating methods could be chained."

That was a side suggestion. I withdraw that part. Chaining methods is
sometimes useful but we can go without it.


pozdrawiam,
Arkadiusz Bulski
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Matt Gilson]

I lurk around here a lot more than I actually post -- But in this case I
find myself weighing in firmly on the side of "please don't add this".
It'd just add noise in the documentation and __builtins__ namespace.  There
are millions of useful functions that *could* be added to the `list` object
(or to the `__builtins__` namespace).  It's the job of the core dev team to
decide which are the most useful for everybody and this is one where (in my
opinion) the usefulness doesn't justify the additional clutter.

It seems like the primary argument for this is that some users would like
to have this functionality baked into an interactive session.  If that's
the case, then I'd like to point out that if you find yourself wanting this
in an interactive session frequently there is an interactive startup file

that you can modify so that this function will always be available to
*you* whenever
you start up an interactive session (and you can put whatever else in there
that you like as well :-).

I hope that helps.

On Wed, Sep 7, 2016 at 3:22 PM, Sven R. Kunze  wrote:

> Maybe, there's a misunderstanding here and I hope I didn't waste too much
> of your and my time.
>
> Not sure where I got this from but my last status was that Arek would like
> to have a "shuffled" function that does exactly what you described. Maybe,
> that doesn't fit the reports description but his pull request will tell the
> truth. :)
>
>
> About the confusion of returning None. It's not confusing in the sense of
> "o my god, I did it wrong, I need to learn it", but more, like "I used
> shuffle, BECAUSE it's the only one I could find in a hurry" and then
> ascertain "why the heck is there no alternative returning the shuffled
> result instead of overwriting my list? I would expect this from Python as
> it already provides both alternatives for sorting".
>
>
> Sven
>
>
>
> On 08.09.2016 00:02, Tim Peters wrote:
>
>> [Sven R. Kunze ]
>>
>>> I am not questioning experience which everyone in a team can benefit
>>> from.
>>>
>>>
>>> BUT experienced devs also need to recognize and respect the fact that
>>> younger/unexperienced developers are just better in detecting
>>> inconsistencies and bloody work-arounds. They simply haven't had to live
>>> with them for so long. Experienced devs just are stuck in a rut/are
>>> routine-blinded: "we've done that for years", "there's no better way".
>>>
>>>
>>> That's the way we do it in our teams. Employing the new guys as some
>>> sort of
>>> inconsistency detectors. This way, they learn to find their way around
>>> the
>>> code base and they can improve it by doing so.
>>>
>>> And I would never allow it in my team, to dismiss this kind of
>>> observation
>>> from new colleagues. It's invaluable as they will become routine-blinded
>>> as
>>> well.
>>>
>> I have been more than willing to discuss it, and I did not close the
>> issue report.  I did say I was opposed to it, but that's simply
>> because I am, and I explained there too _why_ I was opposed.
>>
>> Do you have anything to say about the specific proposal?  I doubt
>> either of us has found this meta-discussion useful.  I'm still looking
>> for a compelling use case.  The only concrete thing anyone has noted
>> in `shuffled()`'s favor so far is that sometimes they're surprised by
>> the behavior of random.shuffle(list) returning None in an interactive
>> shell (noted by you, and by another, and I cheerfully own up to being
>> a bit surprised by that too long ago).   But that's an observation
>> about `random.shuffle()`, not about the proposed `shuffled()`.
>>
>>
>> [...] I would be far more annoyed if, e.g.,

 random.shuffle(some_million_element_list)
>>>
>>   swamped my terminal with mountains of output.

>>> But you readily accept this behavior for "sorted"? That makes no sense at
>>> all.
>>>
>> Of course it does.  The only analogy to random.shuffle(big_list)
>> returning None that makes a lick of sense here is that big_list.sort()
>> also returns None.  IF a `shuffled()` function is introduced, then of
>> course it should return its result - just like `sorted()` returns its
>> result.
>>
>>
>> You can't both behaviors simultaneously, so the status quo wins.
 Indeed, the venerable status quo ;-)

>>> Nobody said to change "shuffle".
>>>
>> A verbatim quote from the first message in this thread:
>>
>>  "Also shuffle() should return self so mutating methods could be
>> chained."
>>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 

[image: pattern-sig.png]

Matt Gilson // SOFTWARE ENGINEER

E: m...@getpattern.com // P: 603.892.7736

We’re looking for beta testers.  Go here
 to sign up!

Re: [Python-ideas] Typecheckers: there can be only one

[Sven R. Kunze]

On 08.09.2016 00:00, Paul Moore wrote:

On 7 September 2016 at 22:31, Hugh Fisher  wrote:

The average programmer such as myself will expect that if I write code
specifying the type of a variable or whatever it should *do
something*. It's code, I wrote it, it should be interpreted.

Reading the documentation should correct that mistaken expectation.


If you need to read the documentation in order to understand a product, 
something's wrong. Especially given the simplicity of the examples.


Some smartphones are delivered without a manual because they are 
considered to be as intuitive as breathing or walking.



Type annotations are code, not tests.

Not in Python they aren't.


Unfortunately, that's not true. :)


Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Typecheckers: there can be only one

[Ivan Levkivskyi]

On 8 September 2016 at 00:00, Paul Moore  wrote:


> > Type annotations are code, not tests.
>
> Not in Python they aren't.
>

Well, to a certain extent. One can try something like this in REPL:

from typing import get_type_hints
import __main__

s: str

class C:
x: int

get_type_hints(C)
get_type_hints(__main__)

Although please remember that the PEP is still provisional and
implementation details may change.

--
Ivan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Sven R. Kunze]

On 07.09.2016 02:49, Chris Kaynor wrote:
I'll weigh in and say that I've had a few cases where I've wanted a 
shuffled function, but not many. The vast majority are interactive 
uses, where I want to get a sampling of data, and in those cases I'm 
normally just printing the output (often, by letting the REPL handle it).


I'm fairly sure I've never wanted a shuffled in the actual code, and 
shuffle is almost always what I want (or just pulling items at random).


Probably the most common case is to produce a list of random numbers 
in a (small) range. The current code looks roughly like:


import random
items = list(range(10))
random.shuffle(items)
items # this is interactive, so this prints it for me

As this does not come up often, I almost invariably write the 
following first:

import random
random.shuffle(range(10))

Then get no output and write the first form. It is not a major 
difference, and only comes up maybe a few times a year at most for me.




That sounds extremely familiar. I would say the interactive session is 
the clear use-case here. But don't ask me why I need a shuffled list of 
somethings (probably for some highly vicious and immoral reasons ;-) ).


@David
Your idea of a PyPI package could almost work. However, in an 
interactive python console, I expect as much batteries included as 
possible to make it as quick as possible. And considering how simple 
such wrapper would be, it almost does not warrant the download of a 
third-party package.


@Tim
Of course it's easy to write a wrapper function. But from what I gather 
here, this is not the point. In interactive sessions, I find it highly 
annoying when I need to define my own functions. Especially because I 
need to do it again in a new python session.


Somebody provided a one-line hack using "sorted" to emulate "shuffled". 
The statement basically is: shuffling is a special kind of sorting. So, 
I would expect the interface to work the same. That at least suffices 
for me to understand Arek's point of view.


I would even go so far as to say:

shuffled(my_list) # returns a new shuffled list
my_list.shuffle() # shuffles in-place

Allowing to plug-in the RNG, when needed.

Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Shuffled

[Tim Peters]

[Sven R. Kunze ]
> I am not questioning experience which everyone in a team can benefit from.
>
>
> BUT experienced devs also need to recognize and respect the fact that
> younger/unexperienced developers are just better in detecting
> inconsistencies and bloody work-arounds. They simply haven't had to live
> with them for so long. Experienced devs just are stuck in a rut/are
> routine-blinded: "we've done that for years", "there's no better way".
>
>
> That's the way we do it in our teams. Employing the new guys as some sort of
> inconsistency detectors. This way, they learn to find their way around the
> code base and they can improve it by doing so.
>
> And I would never allow it in my team, to dismiss this kind of observation
> from new colleagues. It's invaluable as they will become routine-blinded as
> well.

I have been more than willing to discuss it, and I did not close the
issue report.  I did say I was opposed to it, but that's simply
because I am, and I explained there too _why_ I was opposed.

Do you have anything to say about the specific proposal?  I doubt
either of us has found this meta-discussion useful.  I'm still looking
for a compelling use case.  The only concrete thing anyone has noted
in `shuffled()`'s favor so far is that sometimes they're surprised by
the behavior of random.shuffle(list) returning None in an interactive
shell (noted by you, and by another, and I cheerfully own up to being
a bit surprised by that too long ago).   But that's an observation
about `random.shuffle()`, not about the proposed `shuffled()`.


>> [...] I would be far more annoyed if, e.g.,
>>
>> >>> random.shuffle(some_million_element_list)
>>
>>  swamped my terminal with mountains of output.

> But you readily accept this behavior for "sorted"? That makes no sense at
> all.

Of course it does.  The only analogy to random.shuffle(big_list)
returning None that makes a lick of sense here is that big_list.sort()
also returns None.  IF a `shuffled()` function is introduced, then of
course it should return its result - just like `sorted()` returns its
result.


>> You can't both behaviors simultaneously, so the status quo wins.
>> Indeed, the venerable status quo ;-)

> Nobody said to change "shuffle".

A verbatim quote from the first message in this thread:

"Also shuffle() should return self so mutating methods could be chained."
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Typecheckers: there can be only one

[Paul Moore]

On 7 September 2016 at 22:31, Hugh Fisher  wrote:
> The average programmer such as myself will expect that if I write code
> specifying the type of a variable or whatever it should *do
> something*. It's code, I wrote it, it should be interpreted.

Reading the documentation should correct that mistaken expectation.

> Type annotations are code, not tests.

Not in Python they aren't.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Typecheckers: there can be only one

[Hugh Fisher]

On Wed, Sep 7, 2016 at 9:56 PM, Nick Coghlan  wrote:
>
> Exactly - in a dynamic language like Python, running a typechecker is
> a form of testing, rather than a necessary part of getting the code to
> work in the first place.
>

The average programmer such as myself will expect that if I write code
specifying the type of a variable or whatever it should *do
something*. It's code, I wrote it, it should be interpreted.

Assuming PEP 526 is implemented, suppose as a very new programmer I write

foo: dict = 0

(If anyone thinks even new programmers wouldn't write that, you've
never taught an introductory programming course.)

It takes very little programming experience to know that is flat out
wrong. I cannot think of any other programming language with type
notation where this would not be immediately flagged as an error. But
Python will silently accept it? That's the kind of one liner people
put on presentation slides when they want to make fun of how bad
programming languages like JavaScript are.

A more subtle example that came up recently on one of the lists (which
I cannot attribute it properly right now):

c : complex = 1.0

My C compiler is smart enough to figure out it should add an imaginary
zero to the RHS. As an average programmer, I'd expect that Python
would be at least as smart and likewise convert 1.0 into a complex
value. Or it could raise an error saying that the RHS is not a complex
number. Given the code I've written, either makes sense.

What I would not expect is for the interpreter to silently assign a
scalar 1.0 to c and continue. That's just ... WTF?

Type annotations are code, not tests.

-- 

cheers,
Hugh Fisher
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 530: Asynchronous Comprehensions

[Nick Coghlan]

On 8 Sep 2016 02:48, "Guido van Rossum"  wrote:
>
> On Wed, Sep 7, 2016 at 9:34 AM, Yury Selivanov 
wrote:
> > Thank you for accepting the PEP!  Will focus on the implementation now.
>
> You're welcome! Good luck with the implementation.
>
> @Nick, regarding the suggestion to use
>
>   [ for  in async ]
>
> instead of
>
>   [ async for  in ]
>
> I think the lack of grammaticality is actually a feature -- we don't
> want people to confuse `async ` with `await `.

Aye, that problem occurred to me after my last post and is definitely a
concern.

> Python has no other postfix syntax like this (unless you count f(),
> d[k] or x.a as postfix) so I don't think people are likely to mistake
> this for an 'async' postfix on  rather than a prefix on 'for'.
> Hopefully they've seen an 'async for' statement before they encounter
> an async comprehension.

I also realised that the worst case outcome I can see for the PEP 492 based
syntax is some folks avoiding them in favour of the statement version
because they don't like how the expression version reads (ala lambda), and
as worst case scenarios go, that's not a particularly worrying one :)

Cheers,
Nick.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 530: Asynchronous Comprehensions

[Yury Selivanov]

On 2016-09-06 7:19 PM, Guido van Rossum wrote:

On Tue, Sep 6, 2016 at 10:42 AM, Yury Selivanov  wrote:

On 2016-09-04 3:10 AM, Adam Bartoš wrote:

will await be allowed also in the "if" part of comprehensions? And what
about the "in" part? (e.g. if I'm not mistaken, we may have an asynchronous
function returning an asynchronous iterator.)

Yes, awaits will be allowed.  I'll update the PEP.

Hasn't happened yet... I see this PEP as written in a bit of haste and
very close to the 3.6b1 feature freeze (coming weekend). Usually I
wouldn't accept such a hasty PEP, but the ideas in it seem pretty
uncontroversial, and in line with the existing expectations for
async/await.

Yury, if you manage to get a working implementation signed off by one
other core dev (not me) I can accept the PEP provisionally, under the
same conditions as PEP 525.



Thank you for accepting the PEP!  Will focus on the implementation now.

Yury
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 530: Asynchronous Comprehensions

[Nick Coghlan]

On 8 September 2016 at 00:57, Koos Zevenhoven  wrote:
> On Wed, Sep 7, 2016 at 3:27 PM, Nick Coghlan  wrote:
> [...]
>>
>> The new issue that's specific to comprehensions is that the statement
>> form doesn't have the confounding factor of having an expression to
>> the left of it. Thus, the prefix is unambiguous and can be read as
>> modifying the entire statement rather than just the immediately
>> following keyword:
>>
>> async for row in db.execute(...):
>> process(row)
>>
>> It's also pragmatically necessary right now due to the sleight of hand
>> that Yury used in the code generation pipeline to bring in "async
>> def", "async with", and "async for" without a __future__ statement,
>> but without making them full keywords either.
>
> I don't think this issue strictly has anything to do with where that
> "async" is put in the syntax, as long as it's used within the
> definition of an async function:
>
> async def function():
> # in here, is effectively "async" a keyword.

Good point - I was thinking there was additional cleverness around
"async for" and "async with" as well, but you're right that it's
specifically "async def" that enables the pseudo-keyword behaviour.

> [and Nick writes:]
>> [process(row) async for row in db.execute(...)]
>>
>> When reading that, is "async" a postfix operator being used in a
>> normal comprehension (wrong, but somewhat plausible)? Or is it part of
>> a compound keyword with "for" that modifies the iteration behaviour of
>> that part of the comprehension (the correct interpretation)?
>>
>> [(process(row) async) for row in db.execute(...)]
>> [process(row) (async for) row in db.execute(...)]
>>
>> The postfix operator interpretation is just plain wrong, but even the
>> correct interpretation as a compound keyword sits between two
>> expressions *neither* of which is the one being modified (that would
>> be "db.execute()")
>>
>> By contrast, if the addition of full async comprehensions is deferred
>> to 3.7 (when async becomes a true keyword), then the infix spelling
>> can be permitted in both the statement and comprehension forms:
>
> That's an interesting suggestion. What exactly is the relation between
> deferring this PEP and permitting the infix spelling?

Just a mistake on my part regarding how we were currently handling
"async" within "async def" statements. With that mistake corrected,
there may not be any need to defer the suggestion, since it already
behaves as a keyword in the context where it matters.

That said, we *are* doing some not-normal things in the code
generation pipeline to enable the pseudo-keyword behaviour, so I also
wouldn't be surprised if there was a practical limitation on allowing
the "async" to appear after the "in" rather than before the "for"
prior to 3.7.

It's also worth reviewing the minimalist grammar changes in PEP 492
and the associated discussion about "async def" vs "def async":

* https://www.python.org/dev/peps/pep-0492/#grammar-updates
* https://www.python.org/dev/peps/pep-0492/#why-async-def-and-not-def-async

Changing "for_stmt" to allow the "for TARGET in [ASYNC] expr" spelling
isn't as tidy a modification as just allowing ASYNC in front of any of
def_stmt, for_stmt and with_stmt.

> This would make it more obvious at a first glance, whether something
> is a with statement or for loop. The word "async" there is still not
> very easy to miss, especially with highlighted syntax.
>
> I didn't realize (or had forgotten) that PEP 492 is provisional.

Right, and one of the reasons for that was because we hadn't fully
worked through the implications for comprehensions and generator
expressions at the time.

Now that I see the consequences of attempting to transfer the "async
keyword is a statement qualifier " notion to the expression form, I
think we may need to tweak things a bit :)

>> The beauty of the infix form is that it *doesn't matter* whether
>> someone reads it as a compound keyword with "in" or as a prefix
>> modifying the following expression:
>>
>> [process(row) for row (in async) db.execute(...)]
>> [process(row) for row in (async db.execute(...))]
>>
>> In both cases, it clearly suggests something special about the way
>> "db.execute()" is going to be handled, which is the correct
>> interpretation.
>
> And db.execute is an async iterarable after all, so "async" is a
> suitable adjective for db.execute(...).

Exactly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/