Re: [Python-Dev] Importance of async keyword
On 7 July 2015 at 06:08, Sven R. Kunze srku...@mail.de wrote:
I would like to rewrite/amend it to work asynchronously with minimal effort
such as:
def business_new():
content1 = fork open('big.file').read() # wraps up the calls into
awaitables
content2 = fork open('huge.file').read() # and put them into the event
loop
return content1 + content2 # variables are used = await
evaluation
I might have missed something but I think you get my point.
No, you haven't missed anything, but I think the convenience APIs
we're discussing in this thread are what you need, rather than
async/await.
Specifically, your main application would remain synchronous, but the
event loop could be used to run selected operations in a background
thread:
def business_new():
future1 = asyncio.call_async(open('big.file').read)
future2 = asyncio.call_async(open('huge.file').read)
content1 = asyncio.wait_for_result(future1)
content2 = asyncio.wait_for_result(future2)
return content1 + content2
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 06.07.2015 03:41, Nick Coghlan wrote:
That said, I think there's definitely value in providing a very simple
answer to the how do I make a blocking call from a coroutine?
question, so I filed an RFE to add asyncio.blocking_call:
http://bugs.python.org/issue24571
Nice step forward, Nick. Thanks a lot.
I'm less convinced of the value of asyncio.wait_for_result(), so I
haven't filed an RFE for that one.
I have done that for you, because I feel people need to have a
convenient tool to **bridge both worlds** in either direction:
http://bugs.python.org/issue24578
That is even another issue that came to my mind once in a while but I
forgot to post it here:
How are mature projects are supposed to make the transition to asyncio
when they see performance opportunities in using it?
We have several millions lines of code. I actually imagined we could
simply drop an 'await' once in a while in order to gain from asyncio's
power. As it seems, we need to inconveniently write all sort of wrappers
(unnecessarily from my perspective) to retain existing functionality and
leverage asyncio's strength at the same time in order not to break
anything. That is, in fact, the main reason why I conduct this
discussion. I feel this transition is mostly impossible, very costly or
only possible for new code (where is remains to be seen whether it fits
in the existing codebase).
I also feel I am missing something of the bigger picture and I am not
sure if something like this is planned for the future. But from my
perspective in order to leverage asyncio's power, you need at least two
coroutines running at the same time, right? So, in order to get things
running, I still need some sort of get_event_loop into which I can put
my top-level coroutines.
Assume my venerable business functionality:
def business_old():
content1 = open('big.file').read() # blocks until finished
content2 = open('huge.file').read() # blocks until finished
return content1 + content2
I would like to rewrite/amend it to work asynchronously with minimal
effort such as:
def business_new():
content1 = fork open('big.file').read() # wraps up the calls into
awaitables
content2 = fork open('huge.file').read() # and put them into the
event loop
return content1 + content2 # variables are used =
await evaluation
I might have missed something but I think you get my point.
Correct me if I am wrong, but inferring from the given example of PEP
492, currently, we would need to do the following:
def business_new_2():
content1 = open('big.file').read() # get us two awaitables/futures
content2 = open('huge.file').read() # ...
# somehow the loop magic
loop = asyncio.get_event_loop()
loop.run_until_complete(content1)
loop.run_until_complete(content2)
try:
loop.run_forever() # might be something different
finally:
loop.close()
return content1.result() + content2.result()
I certainly do not want to put that into our codebase. Especially when
this kind of code block could occur at any level many times in various
functions.
Regards,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
Thanks, Nick, for you reasoned response. On 03.07.2015 11:40, Nick Coghlan wrote: On 3 July 2015 at 06:55, Sven R. Kunze srku...@mail.de wrote: My understanding of coloring is needs special treatment. Being special or not (containing an 'await' or not), as long as I don't need to care, I can call them either way (with 'await' or not) and each case works sensibly that's fine with me. I'm afraid you're going to be disappointed in that regard, as wishing that event driven programs behaved more like synchronous programs is like wishing that complex numbers behaved like real numbers. Seems like we stick to this example once again. So, let me get this straight: 1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. 3) I can even add, subtract, multiply and divide complex numbers AND real numbers altogether in a single expression. 4) Granted, complex numbers can do more but that basically follows from their definition and does not jeopardize ease of usage. A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why? Because people think we cannot have the best things of both worlds. D) Granted, awaitables can do more but that basically follows from their definition and does not need to jeopardize ease of usage. There's an extra level of complexity that is being deliberately introduced in order to improve Python's ability to express certain kinds of algorithms, and it isn't effective to just try to wish that complexity away. The payoff is that code that otherwise needs to be written as a long series of disconnected callback chains (as has long been possible with Twisted) can instead be written to look more like comparable synchronous code That is great. So, let's do the next step. (and this approach should bring with it much improved introspection support, at least in 3.5+ now that gi_yieldfrom and cr_await are being exposed at the Python level). Sensible would be something similar to: await function: suspension point and runs the function until completion call awaitable: runs the awaitable until completion These both fail, and deliberately so: we don't know what they're supposed to mean, and we refuse the temptation to guess. Where do we guess here? It is a definition. They're also quite likely to indicate a bug (e.g. forgetting to call a native coroutine function to get the coroutine out of it, forgetting to wait for an awaitable) rather than something folks have done deliberately. It's possible shorthand adapters may emerge over time, like: # Call awaitable from synchronous code def wait_for_result(awaitable): Usage: result = asyncio.wait_for_result(awaitable) return asyncio.get_event_loop().run_until_complete(awaitable.__await__()) # Call blocking operation from asynchronous code def blocking_call(f, *args, **kwds): Usage: result = await asyncio.blocking_call(f, *args, **kwds)) cb = functools.partial(f, *args, **kwds) return asyncio.get_event_loop().run_in_executor(cb) However, those can be written already as utility functions, so we can wait and see how strong the demand is for them as adapters. (They may also be potentially useful to have as recipes in the documentation) Sure, that would be reasonable. You have the first guy asking explicitly for these types of adapters provided by the current syntax and removal of 'async'. Until a solution, we return to watching and have one reason less to switch to Python 3. :-/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Mon, Jul 6, 2015 at 7:50 AM, Sven R. Kunze srku...@mail.de wrote: Seems like we stick to this example once again. So, let me get this straight: 1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. 3) I can even add, subtract, multiply and divide complex numbers AND real numbers altogether in a single expression. 4) Granted, complex numbers can do more but that basically follows from their definition and does not jeopardize ease of usage. Until you run into a TypeError: unorderable types: complex() complex(), at which point you realize that they aren't a simple superset of reals with all the same operations supported. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Sun, Jul 05, 2015 at 11:50:00PM +0200, Sven R. Kunze wrote: Seems like we stick to this example once again. So, let me get this straight: 1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. I don't think that this is a complelling analogy for calling regular functions and awaitable functions. Doing arithmetic is a bit more complicated than just the four elementary operators you mention. Contrast: 10 20 10+20j 20+10j It is not just that complex numbers can do more than real numbers. They can also do *less*. But aside from that, your analogy looks to me like this: (1) I can do arithmetic on reals; (2) I can do arithmetic on complex numbers; (3) and I can do arithmetic on mixed real + complex expressions; (A) I can travel in a car to the shops down the road; (B) I can travel in a space ship to the moon; (C) so why can't I travel in a car to the moon? Or use a space ship to fly to the shops down the road? Because people think we cannot have the best things of both worlds! It seems to me that if people think we cannot have the best things of both worlds (your words), it is because there are solid reasons for that belief. Could somebody build a car that can fly to the moon? Probably, but it would cost *more* than the combination of separate space ship plus car, it would require as much maintenance and support as a space ship, and the fuel economy when driving to work would be terrible. Not to mention all the complaints about the noise and the pollution. I think that the distinction between regular and concurrent routines is practical and necessary, *not* just because of people's closed minds, but because of decades of collective experience with them. To convince me differently, you will need more than some rather dubious analogies or arguments from theoretical purity that sequential syncronous code is just a special case of concurrent asyncronous code. An actual working implementation speaks most loudly of all, but at the very least you will need to stick to concrete arguments, not analogies. Are you aware of any other languages which have eliminated the distinction between regular and concurrent functions? If it has already been done, that would make a good argument in favour of your idea. -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 6 July 2015 at 10:27, Chris Angelico ros...@gmail.com wrote:
On Mon, Jul 6, 2015 at 7:50 AM, Sven R. Kunze srku...@mail.de wrote:
Seems like we stick to this example once again. So, let me get this
straight:
1) I can add, subtract, multiply and divide real numbers.
2) I can add, subtract, multiply and divide complex numbers.
3) I can even add, subtract, multiply and divide complex numbers AND real
numbers altogether in a single expression.
4) Granted, complex numbers can do more but that basically follows from
their definition and does not jeopardize ease of usage.
Until you run into a TypeError: unorderable types: complex()
complex(), at which point you realize that they aren't a simple
superset of reals with all the same operations supported.
Exactly. While complex numbers are a superset of the real numbers from
a value perspective, from a behavioural perspective, there are things
you can do when working only with real numbers that you can't do when
you have to account for the fact that here might be complex numbers
are in the mix. In a Python context, the essential operations specific
to real numbers are listed at
https://docs.python.org/3/library/numbers.html#numbers.Real
There's also a difference between the scope of the math and cmath modules:
import math, cmath
set(dir(math)) - set(dir(cmath))
{'floor', 'pow', 'erf', 'trunc', 'copysign', 'expm1', 'ldexp',
'fsum', 'erfc', 'lgamma', 'frexp', 'gamma', 'factorial', 'log2',
'fabs', 'log1p', 'atan2', 'hypot', 'modf', 'radians', 'degrees',
'fmod', 'ceil'}
It's a general principle of design that expanding the kinds of values
you accept (e.g. from real numbers to complex numbers) means you
reduce the kinds of operations you can assume will work (e.g. to the
behaviours of 2D complex numbers, rather than the 1D real number
line). Similarly, as you step further down the numeric tower from real
numbers to rationals and integers, you reduce the set of supported
values, but you also increase the number of defined behaviours.
When it comes to coroutines and subroutines, coroutines are the
superset - a subroutine is just a coroutine that never suspends before
producing a result. You can thus make certain simplifying assumptions
when executing subroutines that you can't make when executing a
coroutine.
It also turns out that never suspends is actually problematic, which
is why nominally subroutine based code these days tends to instead
have *implicit* suspension points based on either operating system
level pre-emptive multithreading where suspension may occur at any
point or greenlet style state switching where suspension may occur as
part of any function call (whether an explicit call or as part of a
syntactic protocol). These approaches provide parallel execution at
the expense of the ability to reason locally about code correctness,
which then causes all sorts of *other* problems.
That said, I think there's definitely value in providing a very simple
answer to the how do I make a blocking call from a coroutine?
question, so I filed an RFE to add asyncio.blocking_call:
http://bugs.python.org/issue24571
I'm less convinced of the value of asyncio.wait_for_result(), so I
haven't filed an RFE for that one.
Regards,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 6 July 2015 at 10:49, Steven D'Aprano st...@pearwood.info wrote: On Sun, Jul 05, 2015 at 11:50:00PM +0200, Sven R. Kunze wrote: Seems like we stick to this example once again. So, let me get this straight: 1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. I don't think that this is a complelling analogy for calling regular functions and awaitable functions. I actually really like the analogy, as the Why can't complex numbers be just like real numbers? reaction is pretty common when folks are first learning to use them for things like signal analysis. I know it didn't really sink in for me at university - it was only a couple of years into doing digital signal processing full time that the concepts involved in switching back and forth between linear real number based time domain analysis and cyclical complex number based frequency domain analysis really started to make sense to me. There's a wonderful page at http://betterexplained.com/articles/a-visual-intuitive-guide-to-imaginary-numbers/ which not only does a great job of providing a relatively intuitive explanation of the behaviour of complex numbers as an answer to the question What is the square root of negative 1?, it also compares them to the original reactions to the absurd notion of negative numbers as an answer to the question What is the result of subtracting a larger number from a smaller one?. Why can't coroutines be just like subroutines? strikes me as being a similar case where the answer is because not everything can be appropriately modelled as a subroutine, but that answer isn't going to make intuitive sense if you've never personally encountered the limits of subroutine based algorithm design. I also think the analogy helps provide good design guidance, as folks are already familiar with the notion of use real numbers if you can, complex numbers if you need to, and extending that to an attitude of use subroutines if you can, coroutines if you need to would be a *very* good thing in terms of encouraging maintainable designs. Are complex numbers better than real numbers? is hopefully a self-evidently nonsensical question - some things are better modelled as complex numbers, others as real numbers, so the only reasonable answer is to ask What are you trying to model?. Are coroutines better than subroutines? is the same kind of question, just applied to algorithm design rather than numeric modelling. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why? Function != awaitable - the answer is right there in the terminology. Different names, different things. Given A, B and the fact that an awaitable is just an object, here's another important point: * I can call a function and might get an awaitable. If the function was decorated with async, the might becomes will, but it's still just a function returning a value. Because an awaitable is indistinguishable from any other value, the compiler doesn't know whether to await or not, so it has to be specified by the developer. If the language specifies it, then it's impossible to handle awaitable objects (for example, to put them in a list and await them later). C# is another language that implemented this exact model a few years ago and it works well. Cheers, Steve Top-posted from my Windows Phone From: Sven R. Kunzemailto:srku...@mail.de Sent: 7/5/2015 14:50 To: python-dev@python.orgmailto:python-dev@python.org Subject: Re: [Python-Dev] Importance of async keyword Thanks, Nick, for you reasoned response. On 03.07.2015 11:40, Nick Coghlan wrote: On 3 July 2015 at 06:55, Sven R. Kunze srku...@mail.de wrote: My understanding of coloring is needs special treatment. Being special or not (containing an 'await' or not), as long as I don't need to care, I can call them either way (with 'await' or not) and each case works sensibly that's fine with me. I'm afraid you're going to be disappointed in that regard, as wishing that event driven programs behaved more like synchronous programs is like wishing that complex numbers behaved like real numbers. Seems like we stick to this example once again. So, let me get this straight: 1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. 3) I can even add, subtract, multiply and divide complex numbers AND real numbers altogether in a single expression. 4) Granted, complex numbers can do more but that basically follows from their definition and does not jeopardize ease of usage. A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why? Because people think we cannot have the best things of both worlds. D) Granted, awaitables can do more but that basically follows from their definition and does not need to jeopardize ease of usage. There's an extra level of complexity that is being deliberately introduced in order to improve Python's ability to express certain kinds of algorithms, and it isn't effective to just try to wish that complexity away. The payoff is that code that otherwise needs to be written as a long series of disconnected callback chains (as has long been possible with Twisted) can instead be written to look more like comparable synchronous code That is great. So, let's do the next step. (and this approach should bring with it much improved introspection support, at least in 3.5+ now that gi_yieldfrom and cr_await are being exposed at the Python level). Sensible would be something similar to: await function: suspension point and runs the function until completion call awaitable: runs the awaitable until completion These both fail, and deliberately so: we don't know what they're supposed to mean, and we refuse the temptation to guess. Where do we guess here? It is a definition. They're also quite likely to indicate a bug (e.g. forgetting to call a native coroutine function to get the coroutine out of it, forgetting to wait for an awaitable) rather than something folks have done deliberately. It's possible shorthand adapters may emerge over time, like: # Call awaitable from synchronous code def wait_for_result(awaitable): Usage: result = asyncio.wait_for_result(awaitable) return asyncio.get_event_loop().run_until_complete(awaitable.__await__()) # Call blocking operation from asynchronous code def blocking_call(f, *args, **kwds): Usage: result = await asyncio.blocking_call(f, *args, **kwds)) cb = functools.partial(f, *args, **kwds) return asyncio.get_event_loop().run_in_executor(cb) However, those can be written already as utility functions, so we can wait and see how strong the demand is for them as adapters. (They may also be potentially useful to have as recipes in the documentation) Sure, that would be reasonable. You have the first guy asking explicitly for these types of adapters provided by the current syntax and removal of 'async'. Until a solution, we return to watching and have one reason less to switch to Python 3. :-/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe
Re: [Python-Dev] Importance of async keyword
On 3 July 2015 at 06:55, Sven R. Kunze srku...@mail.de wrote: My understanding of coloring is needs special treatment. Being special or not (containing an 'await' or not), as long as I don't need to care, I can call them either way (with 'await' or not) and each case works sensibly that's fine with me. I'm afraid you're going to be disappointed in that regard, as wishing that event driven programs behaved more like synchronous programs is like wishing that complex numbers behaved like real numbers. There's an extra level of complexity that is being deliberately introduced in order to improve Python's ability to express certain kinds of algorithms, and it isn't effective to just try to wish that complexity away. The payoff is that code that otherwise needs to be written as a long series of disconnected callback chains (as has long been possible with Twisted) can instead be written to look more like comparable synchronous code (and this approach should bring with it much improved introspection support, at least in 3.5+ now that gi_yieldfrom and cr_await are being exposed at the Python level). Sensible would be something similar to: await function: suspension point and runs the function until completion call awaitable: runs the awaitable until completion These both fail, and deliberately so: we don't know what they're supposed to mean, and we refuse the temptation to guess. They're also quite likely to indicate a bug (e.g. forgetting to call a native coroutine function to get the coroutine out of it, forgetting to wait for an awaitable) rather than something folks have done deliberately. It's possible shorthand adapters may emerge over time, like: # Call awaitable from synchronous code def wait_for_result(awaitable): Usage: result = asyncio.wait_for_result(awaitable) return asyncio.get_event_loop().run_until_complete(awaitable.__await__()) # Call blocking operation from asynchronous code def blocking_call(f, *args, **kwds): Usage: result = await asyncio.blocking_call(f, *args, **kwds)) cb = functools.partial(f, *args, **kwds) return asyncio.get_event_loop().run_in_executor(cb) However, those can be written already as utility functions, so we can wait and see how strong the demand is for them as adapters. (They may also be potentially useful to have as recipes in the documentation) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
My understanding of coloring is needs special treatment. Being special or not (containing an 'await' or not), as long as I don't need to care, I can call them either way (with 'await' or not) and each case works sensibly that's fine with me. Sensible would be something similar to: call function: business as usual await function: suspension point and runs the function until completion call awaitable: runs the awaitable until completion await awaitable: suspension point and could be suspended internally as well On 02.07.2015 00:02, Greg Ewing wrote: Sven R. Kunze wrote: I like the 'await' syntax to mark suspension points. But the 'async' coloring makes no sense to me. It is an implementation details of asyncio (IMHO). Functions containing an await are going to be coloured in any case -- that's unavoidable, given that await is built on the iterator protocol. The only question is whether to have an explicit marker for it. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
Sven R. Kunze wrote: I like the 'await' syntax to mark suspension points. But the 'async' coloring makes no sense to me. It is an implementation details of asyncio (IMHO). Functions containing an await are going to be coloured in any case -- that's unavoidable, given that await is built on the iterator protocol. The only question is whether to have an explicit marker for it. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
Hi, I have never said I wanted implicit asyncio. Explicit is the Python way after all, it served me well and I stick to that. I like the 'await' syntax to mark suspension points. But the 'async' coloring makes no sense to me. It is an implementation details of asyncio (IMHO). From what I can gather, there are 4 distinct cases I would need to take care of. However, I do not wish the team to go through of all of them. Additionally, they would even need to think of whether a function returns an 'awaitable that returns an int', an 'int' or both depending on the input parameters (or due to an implementation error). Maybe, somebody even makes a crazy error like returning an 'awaitable that returns an awaitable that returns an int'. If somebody likes to create/handle this kind of stuff (for whatever reason), using the asyncio library and creating crazy wrapper objects would be the way to go; not the default syntax. Regards, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Sat, Jun 27, 2015 at 12:20 AM, Ethan Furman et...@stoneleaf.us wrote: As Nick said earlier: the caller always blocks; by extension (to my mind, at least) putting an `await` in front of something is saying, it's okay if other tasks run while I'm blocking on this call. Apologies if this is a really REALLY dumb question, but... How hard would it be to then dispense with the await keyword, and simply _always_ behave that way? Something like: def data_from_socket(): # Other tasks may run while we wait for data # The socket.read() function has yield points in it data = socket.read(1024, 1) return transmogrify(data) def respond_to_socket(): while True: data = data_from_socket() # We can pretend that socket writes happen instantly, # but if ever we can't write, it'll let other tasks wait while # we're blocked on it. socket.write(Got it, next please!) Do these functions really need to be aware that there are yield points in what they're calling? I come from a background of thinking with threads, so I'm accustomed to doing blocking I/O and assuming/expecting that other threads will carry on while we wait. If asynchronous I/O can be made that convenient, it'd be awesome. But since it hasn't already been made that easy in every other language, I expect there's some fundamental problem with this approach, something that intrinsically requires every step in the chain to know what's a (potential) block point. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 26 Jun 2015 05:15, Andrew Svetlov andrew.svet...@gmail.com wrote: Another issue that bothers me, is code reuse. Independent from whether the 'async def' makes sense or not, it would not allow us to reuse asyncio functions as if they were normal functions and vice versa (if I understood that correctly). So, we would have to implement things twice for the asyncio world and the classic world. To me, it would be important to use one function in either world where it suits me better. I am uncertain if that makes sense but right now it does to me. Yes, you cannot call async function from synchronous code. There are two worlds: classic and async. Exactly, the sync/async split is inherent in the problem domain. Nobody really likes this consequence of explicitly asynchronous programming models, and this is a good article on why it's annoying: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ However, it's also one of those cases like binary floating point: there are good underlying reasons things are the way they are, but truly coming to grips with them can take a long time, so in the meantime time it's necessary to just accept it as just one of those things (like learning negative or complex numbers for the first time). For explicitly asynchronous programming, Glyph's post at https://glyph.twistedmatrix.com/2014/02/unyielding.html is a decent starting point on that journey. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 2015-06-26 10:31 AM, Chris Angelico wrote: On Sat, Jun 27, 2015 at 12:20 AM, Ethan Furmanet...@stoneleaf.us wrote: As Nick said earlier: the caller always blocks; by extension (to my mind, at least) putting an `await` in front of something is saying, it's okay if other tasks run while I'm blocking on this call. Apologies if this is a really REALLY dumb question, but... How hard would it be to then dispense with the await keyword, and simply _always_ behave that way? Something like: Chris, Sven, if you want this behavior, gevent Stackless Python are your friends. This approach, however, won't ever be merged in CPython (this was discussed plenty of times both on python-ideas and python-dev). There is also a great essay about explicit vs implicit suspension points: https://glyph.twistedmatrix.com/2014/02/unyielding.html Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Sat, 27 Jun 2015 01:10:33 +1000, Chris Angelico ros...@gmail.com wrote: The way I'm seeing it, coroutines are like cooperatively-switched threads; you don't have to worry about certain operations being interrupted (particularly low-level ones like refcount changes or list growth), but any time you hit an 'await', you have to assume a context switch. That's all very well, but I'm not sure it's that big a problem to accept that any call could context-switch; atomicity is already a concern in other cases, which is why we have principles like EAFP rather than LBYL. Read Glyph's article, it explains why: https://glyph.twistedmatrix.com/2014/02/unyielding.html There's clearly some benefit to being able to assume that certain operations are uninterruptible (because there's no 'await' anywhere in them), but are they truly so? Signals can already interrupt something _anywhere_: Yes, and you could have an out of memory error anywhere in your program as well. (Don't do things in your signal handlers, set a flag.) But that doesn't change the stuff Glyph talks about (and Guido talks about) about *reasoning* about your code. I did my best to avoid using threads, and never invested the time and effort in Twisted. But I love programming with asyncio for highly concurrent applications. It fits in my brain :) --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Sat, Jun 27, 2015 at 2:07 AM, R. David Murray rdmur...@bitdance.com wrote: On Sat, 27 Jun 2015 01:10:33 +1000, Chris Angelico ros...@gmail.com wrote: The way I'm seeing it, coroutines are like cooperatively-switched threads; you don't have to worry about certain operations being interrupted (particularly low-level ones like refcount changes or list growth), but any time you hit an 'await', you have to assume a context switch. That's all very well, but I'm not sure it's that big a problem to accept that any call could context-switch; atomicity is already a concern in other cases, which is why we have principles like EAFP rather than LBYL. Read Glyph's article, it explains why: https://glyph.twistedmatrix.com/2014/02/unyielding.html Makes some fair points, but IMO it emphasizes what I'm saying about atomicity. If you really need it, establish it by something other than just having code that naively progresses through. That's what databasing is for, after all. Threading and signals force you to think about concurrency; other models *may* allow you to pretend it doesn't exist. Anyway, that answers my question about why the explicit this can let other things run marker. Thanks all. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 06/26/2015 06:48 AM, Sven R. Kunze wrote: def business(): return complex_calc(5) def business_new() return await complex_calc(10) Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works. Sadly, I have basically no experience in this area -- perhaps that's why Sven's arguments make sense to me. ;) As Nick said earlier: the caller always blocks; by extension (to my mind, at least) putting an `await` in front of something is saying, it's okay if other tasks run while I'm blocking on this call. Maybe we can get there someday. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 25.06.2015 21:11, Andrew Svetlov wrote: Another issue that bothers me, is code reuse. Independent from whether the 'async def' makes sense or not, it would not allow us to reuse asyncio functions as if they were normal functions and vice versa (if I understood that correctly). So, we would have to implement things twice for the asyncio world and the classic world. To me, it would be important to use one function in either world where it suits me better. I am uncertain if that makes sense but right now it does to me. Yes, you cannot call async function from synchronous code. There are two worlds: classic and async. My point is: why does everyone assume that it has to be like that? @Nick Thanks for these links; nice reads and reflect exactly what I think about these topics. Btw. complex numbers basically works the same way (same API) as integers. I would like to see that for functions and awaitables as well. Still, the general assumption is: the sync/async split is inherent in the problem domain. Why? I do not see that inherent split. @Greg I do not like wrappers IF it is just because to have wrappers. We already have 2 types of wrappers: [normal function call] and [function call using await]. Both works like wrappers right in the place where you need them. @Steven Where would the function suspend if there are zero suspension points? It would not. How can you tell what the suspension points *in* the coroutine are from await func()? Let me answer this with a question: How can you tell what the suspension points *in* the coroutine are from async def? Can you show a code snippet of your proposal? By analogy to PEP 0492: def complex_calc(a): if a = 0: return await open(unicode(a)).read() l = await complex_calc(a - 1) r = complex_calc(a - 2) return l + '; ' + r def business(): return complex_calc(5) def business_new() return await complex_calc(10) Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works. Regards, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
Hello, On Sat, 27 Jun 2015 00:31:01 +1000 Chris Angelico ros...@gmail.com wrote: On Sat, Jun 27, 2015 at 12:20 AM, Ethan Furman et...@stoneleaf.us wrote: As Nick said earlier: the caller always blocks; by extension (to my mind, at least) putting an `await` in front of something is saying, it's okay if other tasks run while I'm blocking on this call. Apologies if this is a really REALLY dumb question, but... How hard would it be to then dispense with the await keyword, and simply _always_ behave that way? Something like: def data_from_socket(): # Other tasks may run while we wait for data # The socket.read() function has yield points in it data = socket.read(1024, 1) return transmogrify(data) def respond_to_socket(): while True: data = data_from_socket() # We can pretend that socket writes happen instantly, # but if ever we can't write, it'll let other tasks wait while # we're blocked on it. socket.write(Got it, next please!) Do these functions really need to be aware that there are yield points in what they're calling? I come from a background of thinking with threads, so I'm accustomed to doing blocking I/O and assuming/expecting that other threads will carry on while we wait. If asynchronous I/O can be made that convenient, it'd be awesome. Some say convenient, others say dangerous. Consider: buf = bytearray(MULTIMEGABYTE) In one function you do: socket.write(buf), in another function (coroutine), you keep mutating buf. So, currently in Python you know if you do: socket.write(buf) Then you know it will finish without interruptions for entire buffer. And if you write: await socket.write(buf) then you know there may be interruption points inside socket.write(), in particular something else may mutate it while it's being written. With threads, you always have to assume this last scenario, and do extra effort to protect against it (mutexes and stuff). Note that it's original thinking of Guido, but seeing how it all combines in asyncio (far from being nice consistent way), it's not surprising that some people will keep not understanding how asyncio works/how to use it, while other will grow disappointed by it. Here's my latest disappointment wrt to asyncio usage in MicroPython: http://bugs.python.org/issue24449 -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Sat, Jun 27, 2015 at 12:51 AM, Paul Sokolovsky pmis...@gmail.com wrote: Some say convenient, others say dangerous. Consider: buf = bytearray(MULTIMEGABYTE) In one function you do: socket.write(buf), in another function (coroutine), you keep mutating buf. So, currently in Python you know if you do: socket.write(buf) Then you know it will finish without interruptions for entire buffer. And if you write: await socket.write(buf) then you know there may be interruption points inside socket.write(), in particular something else may mutate it while it's being written. With threads, you always have to assume this last scenario, and do extra effort to protect against it (mutexes and stuff). Hmm. I'm not sure how this is a problem; whether you use 'await' or not, using a mutable object from two separate threads or coroutines is going to have that effect. The way I'm seeing it, coroutines are like cooperatively-switched threads; you don't have to worry about certain operations being interrupted (particularly low-level ones like refcount changes or list growth), but any time you hit an 'await', you have to assume a context switch. That's all very well, but I'm not sure it's that big a problem to accept that any call could context-switch; atomicity is already a concern in other cases, which is why we have principles like EAFP rather than LBYL. There's clearly some benefit to being able to assume that certain operations are uninterruptible (because there's no 'await' anywhere in them), but are they truly so? Signals can already interrupt something _anywhere_: import signal def mutate(sig, frm): ... print(Mutating the global!) ... lst[5] = 0 ... signal.signal(signal.SIGALRM, mutate) Handlers.SIG_DFL: 0 lst = [1, 2, 3, 4, 5, 6, 7, 8, 9] def spin(): ... while True: ... lst[0] += lst[5] ... lst[2] += lst[0] ... lst[4] += lst[6] ... lst[6] -= lst[2] ... if not lst[5]: ... print(Huh???) ... break ... signal.alarm(2) 0 spin() Mutating the global! Huh??? Any time you're using mutable globals, you have to assume that they can be mutated out from under you. Coroutines don't change this, so is there anything gained by knowing where you can't be interrupted by one specific possible context switch? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 06/26/2015 10:31 AM, Chris Angelico wrote: Apologies if this is a really REALLY dumb question, but... How hard would it be to then dispense with the await keyword, and simply _always_ behave that way? Something like: def data_from_socket(): # Other tasks may run while we wait for data # The socket.read() function has yield points in it data = socket.read(1024, 1) return transmogrify(data) def respond_to_socket(): while True: data = data_from_socket() # We can pretend that socket writes happen instantly, # but if ever we can't write, it'll let other tasks wait while # we're blocked on it. socket.write(Got it, next please!) Do these functions really need to be aware that there are yield points in what they're calling? I think yield points is a concept that needs to be spelled out a bit clearer in the PEP 492. It seems that those points are defined by other means outside of a function defined with async def. From the PEP... * It is a SyntaxError to have yield or yield from expressions in an async function. So somewhere in an async function, it needs to await something with a yield in it that isn't an async function. This seems to be a bit counter intuitive to me. Or am I missing something? Regards, Ron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 26 June 2015 at 23:48, Sven R. Kunze srku...@mail.de wrote: @Nick Thanks for these links; nice reads and reflect exactly what I think about these topics. Btw. complex numbers basically works the same way (same API) as integers. Not using complex numbers in Python - coming to grips with what they mean physically. I would like to see that for functions and awaitables as well. This is akin to asking for unification of classes and modules - they're both namespaces so they have a lot of similarities, but the ways in which they're different are by design. Similarly, subroutines and coroutines are not the same thing: https://en.wikipedia.org/wiki/Coroutine#Comparison_with_subroutines They're *related*, but they're still not the same thing. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 06/26/2015 06:48 AM, Sven R. Kunze wrote: def business(): return complex_calc(5) def business_new() return await complex_calc(10) Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works. Assuming async def business_new (to avoid the syntax error), there's no difference between those functions or the one they're calling: * complex_calc returns an awaitable object that, after you've awaited it, will result in an int. * business returns the return value of complex_calc, which is an awaitable object that, after you've awaited it, will result in an int. * business_new returns an awaitable object that, after you've awaited it, will result in an int. In all three of these cases, the result is the same. The fact that the awaitable object returned from any of them is implemented by a coroutine isn't important (in the same way that an iterable object may be implemented by a generator, but it's really irrelevant). The value of the await syntax is when you're doing something interesting, a.k.a. your function is more than delegating directly to another function: async def business_async(): tasks = [complex_calc_async(i) for i in range(10)] results = [await t for t in tasks] await write_to_disk_async(filename, results) return sum(results) Now it's actually useful to be able to await when we choose. Each call to complex_calc_async() could be starting a thread and then suspending until the thread is complete, so we actually start all 10 threads running here before blocking, and then collect the results in the order we started them (and not the order they complete, though I think asyncio has a function to do that). Without the explicit await this would be impossible. The async def also lets us create coroutines consistently even if they don't await anything: if has_version: async def get_version_async(): return VERSION else: async def get_version_async(): return (await get_major_version_async(), await get_minor_version_async()) async def show_version(): print(await get_version_async()) If, like generators, regular functions became coroutines only in the presence of an await, we'd have to do convoluted code to produce the fast get_version_async(), or else make the caller worry about whether they can skip the await. (Also consider calls that cache their result - a coroutine MUST be awaited, but it doesn't have to await anything if it already has the result). (Aside: the _async suffix is based on the convention used in C#, and it's certainly one that I'll be using throughout my async Python code and encouraging in any code that I review. It's the most obvious way to let callers know whether they need to await the function result or not.) Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 06/26/2015 08:47 AM, Steve Dower wrote: On 06/26/2015 06:48 AM, Sven R. Kunze wrote: def business(): return complex_calc(5) def business_new() return await complex_calc(10) Assuming async def business_new (to avoid the syntax error), there's no difference between those functions or the one they're calling: * complex_calc returns an awaitable object that, after you've awaited it, will result in an int. * business returns the return value of complex_calc, which is an awaitable object that, after you've awaited it, will result in an int. * business_new returns an awaitable object that, after you've awaited it, will result in an int. In all three of these cases, the result is the same. The fact that the awaitable object returned from any of them is implemented by a coroutine isn't important (in the same way that an iterable object may be implemented by a generator, but it's really irrelevant). What? Shouldn't 'business_new' return the int? It did await, after all. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 27 June 2015 at 00:31, Chris Angelico ros...@gmail.com wrote: I come from a background of thinking with threads, so I'm accustomed to doing blocking I/O and assuming/expecting that other threads will carry on while we wait. If asynchronous I/O can be made that convenient, it'd be awesome. Folks, it's worth reading through both https://glyph.twistedmatrix.com/2014/02/unyielding.html and http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programming.html It *is* possible to make a language where *everything* is run as a coroutine, and get rid of the subroutine/coroutine distinction that way - a subroutine would *literally* be executed as a coroutine that never waited for anything. The doesn't suspend case could then be handled as an implicit optimisation, rather than as a distinct type, and the *only* way to do IO would be asynchronously through an event loop, rather than through blocking API calls. But that language wouldn't be Python. Python's core execution model is a synchronous procedural dynamically typed one, with everything beyond that, whether object oriented programming, functional programming, asynchronous programming, gradual typing, etc, being a way of managing the kinds of complexity that arise when managing more state, or more complicated algorithms, or more interactivity, or a larger development team. But since it hasn't already been made that easy in every other language, I expect there's some fundamental problem with this approach, something that intrinsically requires every step in the chain to know what's a (potential) block point. As Glyph explains, implicitly switched coroutines are just shared memory threading under another name - when any function call can cause you to lose control of the flow of execution, you have to consider your locking model in much the same way as you do for preemptive threading. For Python 2, this lightweight threads model is available as https://pypi.python.org/pypi/gevent, and there also appears to have been some good recent progress on Python 3 compatibility: https://github.com/gevent/gevent/issues/38 Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 2015-06-26 1:40 PM, Ethan Furman wrote: On 06/26/2015 08:47 AM, Steve Dower wrote: On 06/26/2015 06:48 AM, Sven R. Kunze wrote: def business(): return complex_calc(5) def business_new() return await complex_calc(10) Assuming async def business_new (to avoid the syntax error), there's no difference between those functions or the one they're calling: * complex_calc returns an awaitable object that, after you've awaited it, will result in an int. * business returns the return value of complex_calc, which is an awaitable object that, after you've awaited it, will result in an int. * business_new returns an awaitable object that, after you've awaited it, will result in an int. In all three of these cases, the result is the same. The fact that the awaitable object returned from any of them is implemented by a coroutine isn't important (in the same way that an iterable object may be implemented by a generator, but it's really irrelevant). What? Shouldn't 'business_new' return the int? It did await, after all. business_new should be defined with an 'async' keyword, that's where all the confusion came from: async def business_new(): return await complex_calc(10) Now, business_new() returns a coroutine (which will resolve to the result of complex_calc awaitable), await business_new() will return an int. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
Ethan Furman wrote: On 06/26/2015 08:47 AM, Steve Dower wrote: On 06/26/2015 06:48 AM, Sven R. Kunze wrote: def business(): return complex_calc(5) def business_new() return await complex_calc(10) Assuming async def business_new (to avoid the syntax error), there's no difference between those functions or the one they're calling: * complex_calc returns an awaitable object that, after you've awaited it, will result in an int. * business returns the return value of complex_calc, which is an awaitable object that, after you've awaited it, will result in an int. * business_new returns an awaitable object that, after you've awaited it, will result in an int. In all three of these cases, the result is the same. The fact that the awaitable object returned from any of them is implemented by a coroutine isn't important (in the same way that an iterable object may be implemented by a generator, but it's really irrelevant). What? Shouldn't 'business_new' return the int? It did await, after all. I assumed async def business_new(), rather than some imaginary await in a non-async function will block because I love to create deadlocks in my code feature. Note that blocking prevents *all* coroutines from making progress, unlike threading. When you await all the way to an event loop, it defers the rest of the coroutine until a signal (via a callback) is raised and continues running other coroutines. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 27 June 2015 at 04:06, Ron Adam ron3...@gmail.com wrote: It seems that those points are defined by other means outside of a function defined with async def. From the PEP... * It is a SyntaxError to have yield or yield from expressions in an async function. So somewhere in an async function, it needs to await something with a yield in it that isn't an async function. This isn't the case - it can be async functions and C level coroutines all the way down. Using a generator or other iterable instead requires adaptation to the awaitable protocol (which makes it possible to tap into all the work that has been done for the generator-based coroutines used previously, rather than having to rewrite it to use native coroutines). This isn't very clear in the currently released beta as we made some decisions to simplify the original implementation that we thought would also be OK from an API design perspective, but turned out to pose significant problems once folks actually started trying to integrate native coroutines with other async systems beyond asyncio. Yury fixed those underlying object model limitations as part of addressing Ben Darnell's report of problems attempting to integrate native coroutine support into Tornado (https://hg.python.org/cpython/rev/7a0a1a4ac639). Yury had already updated the PEP to account for those changes, but I've now also added a specific note regarding the API design change in response to beta feedback: https://hg.python.org/peps/rev/0c963fa25db8 Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Fri Jun 26 16:51:13 CEST 2015, Paul Sokolovsky wrote: So, currently in Python you know if you do: socket.write(buf) Then you know it will finish without interruptions for entire buffer. How do you know that? Are you assuming that socket.write is a builtin, rather than a python method? (Not even a python wrapper around a builtin?) Even if that were true, it would only mean that the call itself is processed within a single bytecode ... there is no guarantee that the write method won't release the GIL or call back into python (and thereby allow a thread switch) as part of its own logic. And if you write: await socket.write(buf) then you know there may be interruption points inside socket.write(), in particular something else may mutate it while it's being written. I would consider that external mutation to be bad form ... at least as bad as violating the expectation of an atomic socket.write() up above. So either way, nothing bad SHOULD happen, but it might anyhow. I'm not seeing what the async-coloring actually bought you... -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 26 Jun 2015 10:46, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Sven R. Kunze wrote: So, we would have to implement things twice for the asyncio world and the classic world. Not exactly; it's possible to create a wrapper that takes an async function and runs it to completion, allowing it to be called from sync code. I can't remember offhand, but there's likely something like this already in asyncio. https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_until_complete It's also possible to use a more comprehensive synchronous-to-asynchronous adapter like gevent to call asynchronous code from synchronous code. Going in the other direction (calling sync code from async) uses a thread or process pool: https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On 25.06.2015 04:16, Steven D'Aprano wrote: On Wed, Jun 24, 2015 at 11:21:54PM +0200, Sven R. Kunze wrote: Thanks, Yury, for you quick response. On 24.06.2015 22:16, Yury Selivanov wrote: Sven, if we don't have 'async def', and instead say that a function is a *coroutine function* when it has at least one 'await' expression, then when you refactor useful() by removing the await from it, it stops being a *coroutine function*, which means that it won't return an *awaitable* anymore. Hence the await useful() call in the important() function will be broken. I feared you would say that. Your reasoning assumes that *await* needs an *explicitly declared awaitable*. Let us assume for a moment, we had no async keyword. So, any awaitable needs to have at least 1 await in it. Why can we not have awaitables with 0 awaits in them? I haven't been following the async discussion in detail, but I would expect that the answer is for the same reason that you cannot have a generator function with 0 yields in it. Exactly. That is why I do not stop here. 'async def' guarantees that function always return a coroutine; it eliminates the need of using @asyncio.coroutine decorator (or similar), which besides making code easier to read, also improves the performance. Not to mention new 'async for' and 'async with' statements. Recently, I read Guido's blog about the history of Python and how he eliminated special cases from Python step by step. As I see it, the same could be done here. What is the difference of a function (no awaits) or an awaitable ( 1 awaits) from an end-user's perspective (i.e. the programmer)? The first is syncronous, the second is asyncronous. Correct. Main questions to me here: do I, as a caller, need to care? My answer would be: none. When used the same way, they should behave in the same manner. As long as, we get our nice tracebacks when something went wrong, everything seems find to me. How is that possible? def func(): # Simulate a time consuming calculation. time.sleep(1) return 42 # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = func() I think that one of us is missing something here. As I said, I haven't followed the whole discussion, so it might be me. But on face value, I don't think what you say is reasonable. Ah, wait. That is not what I intended and I completely agree with Guido when he says: I want to be able to *syntactically* tell where the suspension points are in coroutines. http://code.activestate.com/lists/python-dev/135906/ So, it is either: # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = await func() So, from my perspective, no matter, how many (even zero) suspension points there are in func(), the first variant would still be blocking and the second one not. Many programmers (when adhering to classical programming) conceive the world as if they call a function and simply do not care about how it works and what it does. The main thing is, it does what it is supposed to do. Whether this function achieves the goal by working asynchronously or blocking, AND if I, as a programmer, allow the function to work asynchronously, is a completely different matter. That in turn is extremely well reflected by the 'await' keyword (IMHO at least). Another issue that bothers me, is code reuse. Independent from whether the 'async def' makes sense or not, it would not allow us to reuse asyncio functions as if they were normal functions and vice versa (if I understood that correctly). So, we would have to implement things twice for the asyncio world and the classic world. To me, it would be important to use one function in either world where it suits me better. I am uncertain if that makes sense but right now it does to me. One last thing regarding 'async def', that came to my mind recently, is that it compares to that necessity of Java to decorate functions with all possible exceptions that could be raised inside. Thankfully, we do not need to do that in Python. IIRC, it is considered bad practice as it ties modules together tighter as they need to be. If I add a single exception down in the traceback, I need to re-add it everywhere where that function is used. That is a mess regarding clean code. Somehow, 'async def' reminds me of that, too, but maybe, I am missing something here. P.S. This and many other things were discussed at length on the mailing lists, I suggest you to browse through the archives. I can imagine that. As said I went through of some of them, but it could be that I missed some of them as well. Is there a way to search them through (by not using Google)? You can download the mailing list archive for the relevant months, and use your mail client to search them. Thanks Steven. I already found one on the Web:
Re: [Python-Dev] Importance of async keyword
Another issue that bothers me, is code reuse. Independent from whether the 'async def' makes sense or not, it would not allow us to reuse asyncio functions as if they were normal functions and vice versa (if I understood that correctly). So, we would have to implement things twice for the asyncio world and the classic world. To me, it would be important to use one function in either world where it suits me better. I am uncertain if that makes sense but right now it does to me. Yes, you cannot call async function from synchronous code. There are two worlds: classic and async. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Thu, Jun 25, 2015 at 05:55:53PM +0200, Sven R. Kunze wrote: On 25.06.2015 04:16, Steven D'Aprano wrote: On Wed, Jun 24, 2015 at 11:21:54PM +0200, Sven R. Kunze wrote: [...] What is the difference of a function (no awaits) or an awaitable ( 1 awaits) from an end-user's perspective (i.e. the programmer)? The first is syncronous, the second is asyncronous. Correct. Main questions to me here: do I, as a caller, need to care? I think so. If you call a syncronous function when you want something asyncronous, or vice versa, you're going to be disappointed or confused by the result. [...] # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = func() I think that one of us is missing something here. As I said, I haven't followed the whole discussion, so it might be me. But on face value, I don't think what you say is reasonable. Ah, wait. That is not what I intended and I completely agree with Guido when he says: I want to be able to *syntactically* tell where the suspension points are in coroutines. http://code.activestate.com/lists/python-dev/135906/ So, it is either: # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = await func() So, from my perspective, no matter, how many (even zero) suspension points there are in func(), the first variant would still be blocking and the second one not. Where would the function suspend if there are zero suspension points? How can you tell what the suspension points *in* the coroutine are from await func()? Many programmers (when adhering to classical programming) conceive the world as if they call a function and simply do not care about how it works and what it does. The main thing is, it does what it is supposed to do. Concurrency is not classical single-processing programming. Whether this function achieves the goal by working asynchronously or blocking, AND if I, as a programmer, allow the function to work asynchronously, is a completely different matter. That in turn is extremely well reflected by the 'await' keyword (IMHO at least). It might help if you give a concrete example. Can you show a code snippet of your proposal? A single function with zero suspension points being called both asyncronously and syncronously, where does it suspend? It doesn't need to be a big complex function, something simple will do. Another issue that bothers me, is code reuse. Independent from whether the 'async def' makes sense or not, it would not allow us to reuse asyncio functions as if they were normal functions and vice versa (if I understood that correctly). I expect that it will be simple to write a wrapper that converts an ayncronous function to a syncronous one. It's just a matter of waiting until it's done. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
Sven R. Kunze wrote: # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = await func() Using the word blocking this way is potentially confusing. The calling task is *always* blocked until the operation completes. The difference lies in whether *other* tasks are blocked or not. Whether this function achieves the goal by working asynchronously or blocking, AND if I, as a programmer, allow the function to work asynchronously, is a completely different matter. That in turn is extremely well reflected by the 'await' keyword (IMHO at least). Another issue that bothers me, is code reuse. Independent from whether the 'async def' makes sense or not, it would not allow us to reuse asyncio functions as if they were normal functions and vice versa (if I understood that correctly). You understand correctly. If the function is declared async, you must call it with await; if not, you must not. That's not ideal, but it's an unavoidable consequence of the way async/await are implemented. So, we would have to implement things twice for the asyncio world and the classic world. Not exactly; it's possible to create a wrapper that takes an async function and runs it to completion, allowing it to be called from sync code. I can't remember offhand, but there's likely something like this already in asyncio. One last thing regarding 'async def', that came to my mind recently, is that it compares to that necessity of Java to decorate functions with all possible exceptions that could be raised inside. It's a similar situation, but nowhere near as severe. There are only two possibilities, async or not. Once you've converted a function to async, you won't need to change it again on that score. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
Thanks, Yury, for you quick response. On 24.06.2015 22:16, Yury Selivanov wrote: Sven, if we don't have 'async def', and instead say that a function is a *coroutine function* when it has at least one 'await' expression, then when you refactor useful() by removing the await from it, it stops being a *coroutine function*, which means that it won't return an *awaitable* anymore. Hence the await useful() call in the important() function will be broken. I feared you would say that. Your reasoning assumes that *await* needs an *explicitly declared awaitable*. Let us assume for a moment, we had no async keyword. So, any awaitable needs to have at least 1 await in it. Why can we not have awaitables with 0 awaits in them? 'async def' guarantees that function always return a coroutine; it eliminates the need of using @asyncio.coroutine decorator (or similar), which besides making code easier to read, also improves the performance. Not to mention new 'async for' and 'async with' statements. Recently, I read Guido's blog about the history of Python and how he eliminated special cases from Python step by step. As I see it, the same could be done here. What is the difference of a function (no awaits) or an awaitable ( 1 awaits) from an end-user's perspective (i.e. the programmer)? My answer would be: none. When used the same way, they should behave in the same manner. As long as, we get our nice tracebacks when something went wrong, everything seems find to me. Please, correct me if I am wrong. This is also covered in the rationale section of the PEP [2] [1] https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword [2] https://www.python.org/dev/peps/pep-0492/#rationale-and-goals Thanks, Yury P.S. This and many other things were discussed at length on the mailing lists, I suggest you to browse through the archives. I can imagine that. As said I went through of some of them, but it could be that I missed some of them as well. Is there a way to search them through (by not using Google)? Regard, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
On Wed, Jun 24, 2015 at 11:21:54PM +0200, Sven R. Kunze wrote: Thanks, Yury, for you quick response. On 24.06.2015 22:16, Yury Selivanov wrote: Sven, if we don't have 'async def', and instead say that a function is a *coroutine function* when it has at least one 'await' expression, then when you refactor useful() by removing the await from it, it stops being a *coroutine function*, which means that it won't return an *awaitable* anymore. Hence the await useful() call in the important() function will be broken. I feared you would say that. Your reasoning assumes that *await* needs an *explicitly declared awaitable*. Let us assume for a moment, we had no async keyword. So, any awaitable needs to have at least 1 await in it. Why can we not have awaitables with 0 awaits in them? I haven't been following the async discussion in detail, but I would expect that the answer is for the same reason that you cannot have a generator function with 0 yields in it. 'async def' guarantees that function always return a coroutine; it eliminates the need of using @asyncio.coroutine decorator (or similar), which besides making code easier to read, also improves the performance. Not to mention new 'async for' and 'async with' statements. Recently, I read Guido's blog about the history of Python and how he eliminated special cases from Python step by step. As I see it, the same could be done here. What is the difference of a function (no awaits) or an awaitable ( 1 awaits) from an end-user's perspective (i.e. the programmer)? The first is syncronous, the second is asyncronous. My answer would be: none. When used the same way, they should behave in the same manner. As long as, we get our nice tracebacks when something went wrong, everything seems find to me. How is that possible? def func(): # Simulate a time consuming calculation. time.sleep(1) return 42 # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = func() I think that one of us is missing something here. As I said, I haven't followed the whole discussion, so it might be me. But on face value, I don't think what you say is reasonable. P.S. This and many other things were discussed at length on the mailing lists, I suggest you to browse through the archives. I can imagine that. As said I went through of some of them, but it could be that I missed some of them as well. Is there a way to search them through (by not using Google)? You can download the mailing list archive for the relevant months, and use your mail client to search them. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Importance of async keyword
Hi everybody, recently, I stumbled over the new 3.5 release and in doing so over PEP 0492. After careful consideration and after reading many blog posts of various coders, I first would like to thank Yury Selivanov and everybody else who brought PEP 0492 to its final state. I therefore considered usage within our projects, however, still find hazy items in PEP 0492. So, I would like to contribute my thoughts on this in order to either increase my understanding or even improve Python's async capability. In order to do this, I need a clarification regarding the rationale behind the async keyword. The PEP rationalizes its introduction with: If useful() [...] would become a regular python function, [...] important() would be broken. What bothers me is, why should important() be broken in that case? Regards, Sven R. Kunze ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importance of async keyword
Hi Sven, On 2015-06-24 2:14 PM, Sven R. Kunze wrote: Hi everybody, recently, I stumbled over the new 3.5 release and in doing so over PEP 0492. After careful consideration and after reading many blog posts of various coders, I first would like to thank Yury Selivanov and everybody else who brought PEP 0492 to its final state. I therefore considered usage within our projects, however, still find hazy items in PEP 0492. So, I would like to contribute my thoughts on this in order to either increase my understanding or even improve Python's async capability. In order to do this, I need a clarification regarding the rationale behind the async keyword. The PEP rationalizes its introduction with: If useful() [...] would become a regular python function, [...] important() would be broken. What bothers me is, why should important() be broken in that case? Let me first quote the PEP to make the question more obvious. The section at question is: [1]; it explains why async keyword is needed for function declarations. Here's a code example from the PEP: def useful(): ... await log(...) ... def important(): await useful() Sven, if we don't have 'async def', and instead say that a function is a *coroutine function* when it has at least one 'await' expression, then when you refactor useful() by removing the await from it, it stops being a *coroutine function*, which means that it won't return an *awaitable* anymore. Hence the await useful() call in the important() function will be broken. 'async def' guarantees that function always return a coroutine; it eliminates the need of using @asyncio.coroutine decorator (or similar), which besides making code easier to read, also improves the performance. Not to mention new 'async for' and 'async with' statements. This is also covered in the rationale section of the PEP [2] [1] https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword [2] https://www.python.org/dev/peps/pep-0492/#rationale-and-goals Thanks, Yury P.S. This and many other things were discussed at length on the mailing lists, I suggest you to browse through the archives. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
