Re: [Python-ideas] async objects

[Greg Ewing]

Nathaniel Smith wrote:

It wasn't that we created these keywords to solve some
implementation problem and then inflicted them on users.


I disagree -- looking at the history of how we
ended up with async/await, it looks to me like
this is exactly what *did* happen.

First we had generators. Then 'yield from' was
invented to (among other things) leverage them as
a way of getting lightweight threads. Then 'await'
was introduced as a nicer way to spell 'yield from'
when using it for that purpose.

Saying that 'await' is good for you because it
makes the suspension points visible seems to me
a rationalisation after the fact. It was something
that emerged from the implementation, not a
prior design requirement.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 06 Oct 2016, at 07:15, Stephen J. Turnbull 
>  wrote:
> 
> Nick Coghlan writes:
> 
>> Python's core runtime model is the C runtime model: threads (with a
>> local stack and access to a global process heap) and processes (which
>> contain a heap and one or more threads). Anything else we do (whether
>> it's generators, coroutines, or some other form of paused execution
>> like callback management) gets layered on top of that runtime model.
>> When folks ask questions like "Why can't Python be more like Go?",
>> "Why can't Python be more like Erlang?", or "Why can't Python be more
>> like Rust?" and get a negative response, it's usually because there's
>> an inherent conflict between the C runtime model and whatever piece of
>> the Go/Erlang/Rust runtime model we want to steal.
> 
> How can there be a conflict between Python implementing the C runtime
> model *itself* which says "you can do anything anywhere anytime", and
> some part of Python implementing the more restricted models that allow
> safe concurrency?  If you can do anything, well, you can voluntarily
> submit to compiler discipline to a restricted set.  No?  So it must be
> that the existing constructions (functions, for, with) that need an
> "async" marker have an implementation that is itself unsafe.  This
> need is not being explained very well.  What is also not being
> explained is what would be lost by simply using the "safe"
> implementations generated by the async versions everywhere.

Agree, well put. The Erlang runtime (VM) is also written in C, so anything 
should be possible. 

I do not advocate that Python should be a “new” Erlang or Go, just saying that 
since we are introducing some level of concurrency in Python that we look at 
some of the elegant ways others have achieved this and try to implement 
something like that in Python.

> These may be hard to explain, and I know you, Yury, and Guido are very
> busy.  But it's frustrating for all to see this go around in a circle:
> "it's like it is because it has to be that way, so that's the way it is”.

I understand that there is a lot of backwards compatibility, especially in 
regards to the Python/C interface, but I think that it is possible to find an 
elegant solution to this.

> There's also the question of "is async/await really a language
> feature, or is it patching up a deficiency in the CPython
> implementation that other implementations don't necessarily have?" 
> (which has been brought up before, in less contentious terms).
> 
>> So the "async" keyword in "async def", "async for" and "async with" is
>> essentially a marker saying "This is not a C-like runtime concept
>> anymore!"
> 
> That's understood, of course.  The question that isn't being answered
> well is "why can't that non-C-like runtime concept be like Go or
> Erlang or Rust?"  Or, less obtusely, "what exactly is the 'async'
> runtime concept, and why is it preferred to the concepts implemented
> by Go or Erlang or Rust or gevent or greenlets or Stackless?”

This would be very interesting to understand.

> I guess the answer to "why not Stackless?" is buried in the archives
> for Python-Dev somewhere, but I need to get back to $DAYJOB, maybe
> I'll look it up later.

I will try to look for that, I have some time on my hands, not sure I have have 
the %BRAINSKILL, but never the less…

br
/Rene

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Nathaniel Smith]

On Thu, Oct 6, 2016 at 12:45 AM, Greg Ewing  wrote:
> Nathaniel Smith wrote:
>>
>> It wasn't that we created these keywords to solve some
>> implementation problem and then inflicted them on users.
>
>
> I disagree -- looking at the history of how we
> ended up with async/await, it looks to me like
> this is exactly what *did* happen.
>
> First we had generators. Then 'yield from' was
> invented to (among other things) leverage them as
> a way of getting lightweight threads. Then 'await'
> was introduced as a nicer way to spell 'yield from'
> when using it for that purpose.
>
> Saying that 'await' is good for you because it
> makes the suspension points visible seems to me
> a rationalisation after the fact. It was something
> that emerged from the implementation, not a
> prior design requirement.

I wasn't trying to write a detailed account of the development, as
much as try to capture some essential features. Myth, not history :-).

In the final design, the one and only thing that distinguishes
async/await from gevent is that in the former the suspension points
are visible, and in the latter they aren't. I don't really believe
that it's an accident that people put a lot of effort into creating
async/await in this way at a time when gevent already existed and was
widely used in production, and we have historical documents like
Glyph's blog arguing for visible yield points as a motivation for
async/await, but... even if you think it *was* an accident, it hardly
matters at this point. The core distinguishing feature between
async/await and gevent is the visibility of suspension points, so it
might as well be the case that async/await is designed for exactly
those people who want visible suspension points.

(And I didn't say await or visible suspension points are necessarily
"good for you" -- obviously the implicit and explicit interleaving
approaches have trade-offs you'll have to judge for yourself. But
there are some people in some situations who want implicit
interleaving and async/await is there for them.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Nick Coghlan]

On 6 October 2016 at 05:20, Sven R. Kunze  wrote:
> On 05.10.2016 18:06, Nick Coghlan wrote:
>>
>> [runtime matters]
>
>
> I think I understand your point.
>
> I also hope that others and me could provide you with our perspective. We
> see Python not as a C-like runtime but as an abstract modelling language. I
> know that it's different from the point of view of CPython internals,
> however from the outside Python suggests to be much more than a simple
> wrapper around C. Just two different perspectives.

It's not a question that's up for debate - as a point of factual
history, Python's runtime model is anchored in the C runtime model,
and this pervades the entire language design. Simply wishing that
Python's core runtime design was other than it is doesn't make it so.

We can diverge from that base model when we decide there's sufficient
benefit in doing so (e.g. the object model, the import system, the
numeric tower, exception handling, lexical closures, generators,
generators-as-coroutines, context management, native coroutines), but
whatever we decide to do still needs to be expressible in terms of
underlying operating system provided C primitives, or CPython can't
implement it (and if CPython can't implement a feature as the
reference implementation, that feature can't become part of the
language definition).

Postponing the point at which folks are confronted by those underlying
C-level constraints is often an admirable goal, though - the only
thing that isn't possible without fundamentally changing the language
is getting rid of them entirely.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "equal" builtin function

[Paul Moore]

On 6 October 2016 at 14:45, Filipp Bakanov  wrote:
> For now there are many usefull builtin functions like "any", "all", etc. I'd
> like to propose a new builtin function "equal". It should accept iterable,
> and return True if all items in iterable are the same or iterable is emty.
> That's quite popular problem, there is a discussion of how to perform it on
> stackoverflow
> (http://stackoverflow.com/questions/3844801/check-if-all-elements-in-a-list-are-identical)
> - all suggestions are either slow or not very elegant.
> What do you think about it?

It's not a problem I've needed to solve often, if at all (in
real-world code). But even if we assume it is worth having as a
builtin, what would you propose as the implementation? The
stackoverflow discussion highlights a lot of approaches, all with
their own trade-offs. One problem with a builtin is that it would have
to work on all iterables, which is likely to preclude a number of the
faster solutions (which rely on the argument being an actual list).

It's an interesting optimisation problem, and the discussion gives
some great insight into how to micro-optimise an operation like this,
but I'd question whether it needs to be a language/stdlib feature.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Nick Coghlan]

On 6 October 2016 at 15:15, Stephen J. Turnbull
 wrote:
> Nick Coghlan writes:
>
>  > Python's core runtime model is the C runtime model: threads (with a
>  > local stack and access to a global process heap) and processes (which
>  > contain a heap and one or more threads). Anything else we do (whether
>  > it's generators, coroutines, or some other form of paused execution
>  > like callback management) gets layered on top of that runtime model.
>  > When folks ask questions like "Why can't Python be more like Go?",
>  > "Why can't Python be more like Erlang?", or "Why can't Python be more
>  > like Rust?" and get a negative response, it's usually because there's
>  > an inherent conflict between the C runtime model and whatever piece of
>  > the Go/Erlang/Rust runtime model we want to steal.
>
> How can there be a conflict between Python implementing the C runtime
> model *itself* which says "you can do anything anywhere anytime", and
> some part of Python implementing the more restricted models that allow
> safe concurrency?

Anything is possible in C, but not everything is readily supportable :)

When you design a new language and runtime from scratch, you get to
set new rules and expectations if you want to do that. Ericsson did it
with Erlang and BEAM (the reference Erlang VM) by declaring
"Everything's an Actor in the 'Actor Model' sense, and Actors can send
messages to each other's mailboxes". That pushes you heavily towards
application designs where each "process" is a Finite State Machine
with state changes triggered by external events, or by messages from
other processes. If BEAM had been published as open source a decade
earlier than it eventually was, I suspect the modern computing
landscape would look quite different from the way it does today.

Google did something similar with Golang and goroutines by declaring
that Communicating Sequential Processes would be their core
concurrency primitive rather than C's shared memory threading.

By contrast, Python, C++, Java, C#, Objective-C all retained C's core
thread-based "private stack, shared heap" concurrency model, which
later expanded to also include thread local heap storage. Rust
actually retains this core "private stack, private heap, shared heap"
model, but changes the management of data ownership to avoid the messy
problems that arise in practice when using the "everything is
accessible to every thread by default" model.

> If you can do anything, well, you can voluntarily
> submit to compiler discipline to a restricted set.  No?  So it must be
> that the existing constructions (functions, for, with) that need an
> "async" marker have an implementation that is itself unsafe.

Correct (for a given definition of unsafe): in normal operation,
CPython uses the *C stack* to manage the Python frame stack, so when
you descend into a new function call in CPython, you're also using up
more C level stack space. This means that when CPython throws
RecursionError, what it's actually aiming to prevent is a C level
segfault arising from running out of stack space to manage frames:

  $ ./python -X faulthandler
  Python 3.6.0b1+ (3.6:b995b1f52975, Sep 22 2016, 01:19:04)
  [GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> def f(): f()
  ...
  >>> f()
  Traceback (most recent call last):
File "", line 1, in 
File "", line 1, in f
File "", line 1, in f
File "", line 1, in f
[Previous line repeated 995 more times]
  RecursionError: maximum recursion depth exceeded
  >>> import sys
  >>> sys.setrecursionlimit(int(1e5))
  >>> f()
  Fatal Python error: Segmentation fault

  Current thread 0x7fe977a7c700 (most recent call first):
File "", line 1 in f
File "", line 1 in f
File "", line 1 in f
[]
...
  Segmentation fault (core dumped)

Loops, with statements and other magic method invocations all work
that way - they make a C level call to the magic method implementation
which may end up running a new invocation of the eval loop to evaluate
the bytecode of a magic method implementation that's written in
Python.

The pay-off that CPython gets from this is that we get to delegate
99.9% of the work for supporting different CPU architectures to C
compiler developers, and we get a lot of capabilities "for free" when
it comes to stack management.

The downside is that C runtimes don't officially support swapping out
the stack of the current thread with new contents. It's *possible* to
do that (hence Stackless and gevent), but you're on your own when it
comes to debugging it when it breaks.

That makes it a good candidate for an opt-in "expert users only"
capability - folks that decide gevent is the right answer for their
needs can adopt it if they want to (perhaps restricting their choice
of target platform and C extension modules as a result), while we (as
in the CPython core devs) don't need to keep custom 

[Python-ideas] Add "equal" builtin function

[Filipp Bakanov]

For now there are many usefull builtin functions like "any", "all", etc.
I'd like to propose a new builtin function "equal". It should accept
iterable, and return True if all items in iterable are the same or iterable
is emty.
That's quite popular problem, there is a discussion of how to perform it on
stackoverflow (
http://stackoverflow.com/questions/3844801/check-if-all-elements-in-a-list-are-identical)
- all suggestions are either slow or not very elegant.
What do you think about it?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Greg Ewing]

Nathaniel Smith wrote:

The core distinguishing feature between
async/await and gevent is the visibility of suspension points, so it
might as well be the case that async/await is designed for exactly
those people who want visible suspension points.


They're not quite independent axes, though. Gevent is based
on greenlet, which relies on some slightly dubious tricks at
the C level and doesn't play well with some external libraries.

As far as I know, there's no current alternative that's just
as efficient and portable as asyncio but without the extra
keywords. If you want the full benefits of asyncio, you're
forced to accept explicit suspension points.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] str(slice(10)) should return "slice(10)"

[Neil Girdhar]

Currently str(slice(10)) returns "slice(None, 10, None)"

If the start and step are None, consider not emitting them.  Similarly 
slice(None) is rendered slice(None, None, None).

When you're printing a lot of slices, it's a lot of extra noise.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Greg Ewing]

Nick Coghlan wrote:

When a language usage pattern is supported for that long, but folks
still don't grok how it might benefit them, you have a UX problem, and
one of the ways to address it is to take the existing pattern and give
it dedicated syntax, which is exactly what PEP 492 did.


However, it was just replacing one way of explicitly
marking suspension points ("yield from") with another
("await"). The fact that suspension points are explicitly
marked was driven by the implementation from the beginning.

When I first proposed "yield from" as an aid to using
generators as coroutines, my intention was always to
eventually replace it with something else. PEP 3152 was
my proposal for what the something else might be.
I initially regarded it as a wart that it still
required a special syntax for suspendable calls, and
felt the need to apologise for that.

I was totally surprised when people said they actually
*liked* the idea of explicit suspension points.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Greg Ewing]

Yury Selivanov wrote:
To start, no matter how exactly you want to approach this, it would 
require us to do a *complete rewrite* of CPython internals.  This is so 
complex that we wouldn't be able to even estimate how long it would take 
us.


You could ask the author of Stackless -- he did exactly that
quite a while back.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "equal" builtin function

[Sjoerd Job Postmus]

On Thu, Oct 06, 2016 at 03:01:36PM +0100, Paul Moore wrote:
> On 6 October 2016 at 14:45, Filipp Bakanov  wrote:
> > For now there are many usefull builtin functions like "any", "all", etc. I'd
> > like to propose a new builtin function "equal". It should accept iterable,
> > and return True if all items in iterable are the same or iterable is emty.
> > That's quite popular problem, there is a discussion of how to perform it on
> > stackoverflow
> > (http://stackoverflow.com/questions/3844801/check-if-all-elements-in-a-list-are-identical)
> > - all suggestions are either slow or not very elegant.
> > What do you think about it?
> 
> It's not a problem I've needed to solve often, if at all (in
> real-world code). But even if we assume it is worth having as a
> builtin, what would you propose as the implementation? The
> stackoverflow discussion highlights a lot of approaches, all with
> their own trade-offs. One problem with a builtin is that it would have
> to work on all iterables, which is likely to preclude a number of the
> faster solutions (which rely on the argument being an actual list).
> 
> It's an interesting optimisation problem, and the discussion gives
> some great insight into how to micro-optimise an operation like this,
> but I'd question whether it needs to be a language/stdlib feature.
> 
> Paul

I've needed it several times, but can't really remember what for
anymore, which makes me think it's not really that important.

A motivating reason for adding it to the builtins would be that it can
be written in C instead of Python, and hence be a lot faster. The single
slowest solution is actually the fastest when the difference is detected
very soon (case s3), all others are `O(n)` and not `O(first-mismatch)`.

Though, that means it could also be written in C and provided to PyPI,
at the cost of asking others to install an extra package.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "equal" builtin function

[אלעזר]

It is a real problem. People are used to write `seq == [1, 2, 3]` and it
passes unnoticed (even with type checkers) that if seq changes to e.g. a
tuple, it will cause subtle bugs. It is inconvenient to write `len(seq) ==
3 and seq == [1, 2, 3]` and people often don't notice the need to write it.

(I'd like to note that it makes sense for this operation to be written as

*iter1 == *lst

although it requires a significant change to the language, so a
Sequence.equal function makes sense)

Elazar

On Thu, Oct 6, 2016 at 5:02 PM Paul Moore  wrote:

> On 6 October 2016 at 14:45, Filipp Bakanov  wrote:
> > For now there are many usefull builtin functions like "any", "all", etc.
> I'd
> > like to propose a new builtin function "equal". It should accept
> iterable,
> > and return True if all items in iterable are the same or iterable is
> emty.
> > That's quite popular problem, there is a discussion of how to perform it
> on
> > stackoverflow
> > (
> http://stackoverflow.com/questions/3844801/check-if-all-elements-in-a-list-are-identical
> )
> > - all suggestions are either slow or not very elegant.
> > What do you think about it?
>
> It's not a problem I've needed to solve often, if at all (in
> real-world code). But even if we assume it is worth having as a
> builtin, what would you propose as the implementation? The
> stackoverflow discussion highlights a lot of approaches, all with
> their own trade-offs. One problem with a builtin is that it would have
> to work on all iterables, which is likely to preclude a number of the
> faster solutions (which rely on the argument being an actual list).
>
> It's an interesting optimisation problem, and the discussion gives
> some great insight into how to micro-optimise an operation like this,
> but I'd question whether it needs to be a language/stdlib feature.
>
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "equal" builtin function

[Sjoerd Job Postmus]

On Thu, Oct 06, 2016 at 02:45:11PM +, אלעזר wrote:
> It is a real problem. People are used to write `seq == [1, 2, 3]` and it
> passes unnoticed (even with type checkers) that if seq changes to e.g. a
> tuple, it will cause subtle bugs. It is inconvenient to write `len(seq) ==
> 3 and seq == [1, 2, 3]` and people often don't notice the need to write it.
> 
> (I'd like to note that it makes sense for this operation to be written as
> 
> *iter1 == *lst
> 
> although it requires a significant change to the language, so a
> Sequence.equal function makes sense)
> 
> Elazar
> 

I think you're mistaken about the suggestion. It's not about a function

def equal(it1: Iterable, it2: Iterable) -> bool:

but about a function

def equal(it: Iterable) -> bool:

.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add "equal" builtin function

[אלעזר]

On Thu, Oct 6, 2016 at 5:53 PM Sjoerd Job Postmus 
wrote:

> On Thu, Oct 06, 2016 at 02:45:11PM +, אלעזר wrote:
> > It is a real problem. People are used to write `seq == [1, 2, 3]` and it
> > passes unnoticed (even with type checkers) that if seq changes to e.g. a
> > tuple, it will cause subtle bugs. It is inconvenient to write `len(seq)
> ==
> > 3 and seq == [1, 2, 3]` and people often don't notice the need to write
> it.
> >
> > (I'd like to note that it makes sense for this operation to be written as
> >
> > *iter1 == *lst
> >
> > although it requires a significant change to the language, so a
> > Sequence.equal function makes sense)
> >
> > Elazar
> >
>
> I think you're mistaken about the suggestion.


You are right of course. Sorry.

Elazar
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/