Re: [Python-ideas] Add hooks to asyncio lifecycle
On Tue, Jun 12, 2018 at 5:34 AM Michel Desmoulin wrote: [..] > But even if I have spoken up then, my experience with Python-idea is that most of the time, people would have told me "no". They would have told me that "nobody is going to do that". > I'm getting used to it. It took me a lot of time of "path should inherit from strings" to finally have enough people involved that we get a solution to the problem (which ended up being __fspath__). At the beginning, the answer was "no, there is no problem". > So basically, now I speak up, knowing that people will say "no". But at least I said it. Maybe in 2 years we will go back to that an implement it, or another solution to this problem. [..] > Locking the policy is providing a clean and easy way to do things. Michel, I know you're an avid asyncio user. I've seen your kind comments about asyncio and uvloop on reddit, twitter, etc. So I really don't want to discourage you from posting here and proposing ideas. But arguments like 'people would have told me "no" ... I'm getting used to it.' aren't helpful. I've repeatedly asked you in this thread to provide a couple of good and easy to follow examples so that we can make an informed decision about whether we should add policy locking mechanism or not. Example: We want to create library "A" because it will work only with event loop "B". Checking the event loop once in A's APIs doesn't work [because ...]. Another example: We use asyncio in production and we see that people change policies at runtime all the time; we need to guard against that somehow. Unless you can clearly demonstrate that locking solves real world problems we are not going to implement it, because get_event_loop() and policies are *already* too complicated. You say that Django performs a bunch of sanity checks, but I don't see a good explanation of why you can't just call `isinstance(asyncio.get_event_loop_policy())` in your framework/application. I also keep repeating that libraries, in general, shouldn't be hardcoded to work with some specific policies, and they should not provide alternative policies unless they provide an alternative event loop implementation. And I don't want to add locking to somehow encourage libraries to use policies. I use a bunch of asyncio libraries in my code and none of them (except aiohttp) uses policies. And aiohttp provides policies only to help you run HTTP servers under gunicorn etc, it's up to the user if they want to use them or not. I'm currently prototyping nurseries/cancel scopes for asyncio and I don't need to use policies to make them work. So I honestly don't see a clear case to add locking so far. Maybe having a PEP would allow us to build a compelling case for adding policies locking. Yury ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
> > I still don't get it... If you have a framework you presumably have an > entry point. Why can't you set up your policy in that entrypoint? Why > would a user attempt to change the policy at runtime (you haven't > listed examples of libraries that do this)? Not at run time. But several libraries have several entry points, that you must use in proper order, and setup correctly. Locking helps with finding when you do an improper setup without reading all documentation for all libs completely to make sure you didn't miss a corner case. It will also help project owners because people will come to the bug tracker saying "I have LockedPolicyError" when I do that, which is a lot easier to solve than "I have an AttributeError on a None object". I see a lot of "I want to > protect users from ..." arguments but I haven't yet seen "this and > that happened in production and we haven't been able to debug what > happened for a while". It's like with asyncio.get_event_loop. We haven't seen the problem for a a long time. Then eventually, people starting to use several loops in several threads, and they said that it caused a problem. It's the same here. I'm advising that we don't wait for people to have the problem to solve it. > > Do you handle cases when people install a blocking logging handler in > their async application? We can't prevent that without a great cost. It's terribly complicated problem, especially since logging is blocking and thread safe by design. There is not even good documentation on the best practice on using logging with asyncio. Most people have no idea how to do it (although in my experience, most dev use the logging module incorrectly outside of asyncio too). Twisted rewrote an entire logging system to solve that problem. Locking the policy is not a great cost. And it's about creating a user friendly API. Do you handle cases when a malfunctioning > sys.excepthook is installed? When I override sys.excepthook, I take great care of backuping the old hook, and calling it in my hook. And I really wish the stdlib had something built in to do that cleanly, because I'm pretty sure most lib overriding sys.excepthook all do that in a different way, if they do it at all. Every time I use something to help with stack trace (coloring, auto logging, etc), I have to read the source code to check they are compatible. I should not have to do that. What about cases when users accidentally > import gevent somewhere in their asyncio application and it > monkeypatches the 'socket' module (this is a real horror story, by the > way)? Monkey patching is not officially supported by anything, anywhere, so this argument is moot. But it opens currently another door for my argument: currently the only way to make sure nobody erase our policy is to monkey patch the function to set the policy. Do we really want monkey patch to be the only solution to this problem ? My point is that there are so many things that users can do > that will break any framework, be it asyncio or django or trio. Yes, eventually the question is how easily and cleanly can we provide a solution to avoid the problem or help to debug it. Django is a good example. I run sanity checks on common errors on your model on startup to ease your life as a dev, and the life on the maintainers and their bug tracker. > > This sounds like "if something can happen it will happen" kind of > thing, The reason I bring get_event_loop on the table is that I knew years ago when I read that source code that it would come back to bite us. And I said nothing because I assume the core devs would have though of that and that I was mistaken. But even if I have spoken up then, my experience with Python-idea is that most of the time, people would have told me "no". They would have told me that "nobody is going to do that". I'm getting used to it. It took me a lot of time of "path should inherit from strings" to finally have enough people involved that we get a solution to the problem (which ended up being __fspath__). At the beginning, the answer was "no, there is no problem". So basically, now I speak up, knowing that people will say "no". But at least I said it. Maybe in 2 years we will go back to that an implement it, or another solution to this problem. but I haven't yet seen good examples of real code that suffers > from non-locked policies. Using the nurseries example doesn't count, > as this is something that we want to have as a builtin functionality > in 3.8. So in we will be able to use that in 2 years. > > Locking policies can lead to more predictable user experience; OTOH > what happens if, say, aiohttp decides to lock its policy to use uvloop > and thus make it impossible for its users to use tokio or some other > loop implementation? That would be very, very good. It would make explicit that aiohttp is: - using a custom policy - assuming it's the only one - not allowing another lib to use one - rely on it to work - has
Re: [Python-ideas] Add hooks to asyncio lifecycle
Well, if we need something -- locking is better than hooks. But yes, we need a real life example. While the example is absent let's postpone the solution. On Mon, Jun 11, 2018 at 7:50 PM Yury Selivanov wrote: > > aiohttp doesn't depend on event loop implementation but uses public API > only. > > Yeah, I understand. I was using it as an example of what happens if a > popular library like aiohttp decides to lock the policy for whatever > reason. > > To add to my point: event loop policies should only be used to inject > a custom event loop implementation, like uvloop. They shouldn't be > used to add some framework- or library-specific functionality. That's > why I think that locking policies does not make a lot of sense. > > Yury > On Mon, Jun 11, 2018 at 12:40 PM Andrew Svetlov > wrote: > > > > Sorry, smartphone is not my preferred tool. > > aiohttp doesn't depend on event loop implementation but uses public API > only. > > aiohttp test suite allows to check against asyncio, uvloop, and tokio > but it is another story. > > > > On Mon, Jun 11, 2018 at 7:35 PM Andrew Svetlov > wrote: > >> > >> In my mind aiohttp doesn't depend on > >> > >> On Mon, Jun 11, 2018, 19:24 Yury Selivanov > wrote: > >>> > >>> > I want to abstract that from the user, so I tried to put that in a > >>> policy. But that's dangerous since it can be changed at any time, so I > >>> gave up on it and made it explicit. Of course, if the user misses that > >>> in the doc (hopefully, it's an company internal code so they should be > >>> trained), it will be a bummer to debug. > >>> > >>> I still don't get it... If you have a framework you presumably have an > >>> entry point. Why can't you set up your policy in that entrypoint? Why > >>> would a user attempt to change the policy at runtime (you haven't > >>> listed examples of libraries that do this)? I see a lot of "I want to > >>> protect users from ..." arguments but I haven't yet seen "this and > >>> that happened in production and we haven't been able to debug what > >>> happened for a while". > >>> > >>> Do you handle cases when people install a blocking logging handler in > >>> their async application? Do you handle cases when a malfunctioning > >>> sys.excepthook is installed? What about cases when users accidentally > >>> import gevent somewhere in their asyncio application and it > >>> monkeypatches the 'socket' module (this is a real horror story, by the > >>> way)? My point is that there are so many things that users can do > >>> that will break any framework, be it asyncio or django or trio. > >>> > >>> This sounds like "if something can happen it will happen" kind of > >>> thing, but I haven't yet seen good examples of real code that suffers > >>> from non-locked policies. Using the nurseries example doesn't count, > >>> as this is something that we want to have as a builtin functionality > >>> in 3.8. > >>> > >>> Locking policies can lead to more predictable user experience; OTOH > >>> what happens if, say, aiohttp decides to lock its policy to use uvloop > >>> and thus make it impossible for its users to use tokio or some other > >>> loop implementation? > >>> > >>> Yury > >>> > >>> > >>> > >>> On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin > >>> wrote: > >>> > > >>> > I like it. > >>> > > >>> > First, it solves the issue for policies, and let people decide how > they > >>> > want to deal with the problem (drop the lib, subclass the > >>> > policy/factory, etc). > >>> > > >>> > But it also solves the problem for loops, because loops are set by > the > >>> > task factory, and so you can easily check somebody is changing your > loop > >>> > from you locked policy and do whatever you want. > >>> > > >>> > This also solves the problem of: > >>> > > >>> > - task factories > >>> > - event loop life cycle hooks > >>> > > >>> > Indeed, if somebody needs those, he/she can implement a custom loop, > >>> > which can be safe guarded by the policy, which is locked. > >>> > > >>> > It doesn't have the drawback of my proposal of being overly general, > and > >>> > is quite simple to implement. But it does let people get creative > with > >>> > the stack. > >>> > > >>> > > >>> > > >>> > > >>> > ___ > >>> > Python-ideas mailing list > >>> > Python-ideas@python.org > >>> > https://mail.python.org/mailman/listinfo/python-ideas > >>> > Code of Conduct: http://python.org/psf/codeofconduct/ > >>> > >>> > >>> > >>> -- > >>> Yury > >>> ___ > >>> Python-ideas mailing list > >>> Python-ideas@python.org > >>> https://mail.python.org/mailman/listinfo/python-ideas > >>> Code of Conduct: http://python.org/psf/codeofconduct/ > >> > >> -- > >> Thanks, > >> Andrew Svetlov > > > > -- > > Thanks, > > Andrew Svetlov > > > > -- > Yury > -- Thanks, Andrew Svetlov ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/l
Re: [Python-ideas] Add hooks to asyncio lifecycle
> aiohttp doesn't depend on event loop implementation but uses public API only. Yeah, I understand. I was using it as an example of what happens if a popular library like aiohttp decides to lock the policy for whatever reason. To add to my point: event loop policies should only be used to inject a custom event loop implementation, like uvloop. They shouldn't be used to add some framework- or library-specific functionality. That's why I think that locking policies does not make a lot of sense. Yury On Mon, Jun 11, 2018 at 12:40 PM Andrew Svetlov wrote: > > Sorry, smartphone is not my preferred tool. > aiohttp doesn't depend on event loop implementation but uses public API only. > aiohttp test suite allows to check against asyncio, uvloop, and tokio but it > is another story. > > On Mon, Jun 11, 2018 at 7:35 PM Andrew Svetlov > wrote: >> >> In my mind aiohttp doesn't depend on >> >> On Mon, Jun 11, 2018, 19:24 Yury Selivanov wrote: >>> >>> > I want to abstract that from the user, so I tried to put that in a >>> policy. But that's dangerous since it can be changed at any time, so I >>> gave up on it and made it explicit. Of course, if the user misses that >>> in the doc (hopefully, it's an company internal code so they should be >>> trained), it will be a bummer to debug. >>> >>> I still don't get it... If you have a framework you presumably have an >>> entry point. Why can't you set up your policy in that entrypoint? Why >>> would a user attempt to change the policy at runtime (you haven't >>> listed examples of libraries that do this)? I see a lot of "I want to >>> protect users from ..." arguments but I haven't yet seen "this and >>> that happened in production and we haven't been able to debug what >>> happened for a while". >>> >>> Do you handle cases when people install a blocking logging handler in >>> their async application? Do you handle cases when a malfunctioning >>> sys.excepthook is installed? What about cases when users accidentally >>> import gevent somewhere in their asyncio application and it >>> monkeypatches the 'socket' module (this is a real horror story, by the >>> way)? My point is that there are so many things that users can do >>> that will break any framework, be it asyncio or django or trio. >>> >>> This sounds like "if something can happen it will happen" kind of >>> thing, but I haven't yet seen good examples of real code that suffers >>> from non-locked policies. Using the nurseries example doesn't count, >>> as this is something that we want to have as a builtin functionality >>> in 3.8. >>> >>> Locking policies can lead to more predictable user experience; OTOH >>> what happens if, say, aiohttp decides to lock its policy to use uvloop >>> and thus make it impossible for its users to use tokio or some other >>> loop implementation? >>> >>> Yury >>> >>> >>> >>> On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin >>> wrote: >>> > >>> > I like it. >>> > >>> > First, it solves the issue for policies, and let people decide how they >>> > want to deal with the problem (drop the lib, subclass the >>> > policy/factory, etc). >>> > >>> > But it also solves the problem for loops, because loops are set by the >>> > task factory, and so you can easily check somebody is changing your loop >>> > from you locked policy and do whatever you want. >>> > >>> > This also solves the problem of: >>> > >>> > - task factories >>> > - event loop life cycle hooks >>> > >>> > Indeed, if somebody needs those, he/she can implement a custom loop, >>> > which can be safe guarded by the policy, which is locked. >>> > >>> > It doesn't have the drawback of my proposal of being overly general, and >>> > is quite simple to implement. But it does let people get creative with >>> > the stack. >>> > >>> > >>> > >>> > >>> > ___ >>> > Python-ideas mailing list >>> > Python-ideas@python.org >>> > https://mail.python.org/mailman/listinfo/python-ideas >>> > Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >>> >>> -- >>> Yury >>> ___ >>> Python-ideas mailing list >>> Python-ideas@python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> Thanks, >> Andrew Svetlov > > -- > Thanks, > Andrew Svetlov -- Yury ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
Sorry, smartphone is not my preferred tool. aiohttp doesn't depend on event loop implementation but uses public API only. aiohttp test suite allows to check against asyncio, uvloop, and tokio but it is another story. On Mon, Jun 11, 2018 at 7:35 PM Andrew Svetlov wrote: > In my mind aiohttp doesn't depend on > > On Mon, Jun 11, 2018, 19:24 Yury Selivanov > wrote: > >> > I want to abstract that from the user, so I tried to put that in a >> policy. But that's dangerous since it can be changed at any time, so I >> gave up on it and made it explicit. Of course, if the user misses that >> in the doc (hopefully, it's an company internal code so they should be >> trained), it will be a bummer to debug. >> >> I still don't get it... If you have a framework you presumably have an >> entry point. Why can't you set up your policy in that entrypoint? Why >> would a user attempt to change the policy at runtime (you haven't >> listed examples of libraries that do this)? I see a lot of "I want to >> protect users from ..." arguments but I haven't yet seen "this and >> that happened in production and we haven't been able to debug what >> happened for a while". >> >> Do you handle cases when people install a blocking logging handler in >> their async application? Do you handle cases when a malfunctioning >> sys.excepthook is installed? What about cases when users accidentally >> import gevent somewhere in their asyncio application and it >> monkeypatches the 'socket' module (this is a real horror story, by the >> way)? My point is that there are so many things that users can do >> that will break any framework, be it asyncio or django or trio. >> >> This sounds like "if something can happen it will happen" kind of >> thing, but I haven't yet seen good examples of real code that suffers >> from non-locked policies. Using the nurseries example doesn't count, >> as this is something that we want to have as a builtin functionality >> in 3.8. >> >> Locking policies can lead to more predictable user experience; OTOH >> what happens if, say, aiohttp decides to lock its policy to use uvloop >> and thus make it impossible for its users to use tokio or some other >> loop implementation? >> >> Yury >> >> >> >> On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin >> wrote: >> > >> > I like it. >> > >> > First, it solves the issue for policies, and let people decide how they >> > want to deal with the problem (drop the lib, subclass the >> > policy/factory, etc). >> > >> > But it also solves the problem for loops, because loops are set by the >> > task factory, and so you can easily check somebody is changing your loop >> > from you locked policy and do whatever you want. >> > >> > This also solves the problem of: >> > >> > - task factories >> > - event loop life cycle hooks >> > >> > Indeed, if somebody needs those, he/she can implement a custom loop, >> > which can be safe guarded by the policy, which is locked. >> > >> > It doesn't have the drawback of my proposal of being overly general, and >> > is quite simple to implement. But it does let people get creative with >> > the stack. >> > >> > >> > >> > >> > ___ >> > Python-ideas mailing list >> > Python-ideas@python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> Yury >> ___ >> Python-ideas mailing list >> Python-ideas@python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- > Thanks, > Andrew Svetlov > -- Thanks, Andrew Svetlov ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
In my mind aiohttp doesn't depend on On Mon, Jun 11, 2018, 19:24 Yury Selivanov wrote: > > I want to abstract that from the user, so I tried to put that in a > policy. But that's dangerous since it can be changed at any time, so I > gave up on it and made it explicit. Of course, if the user misses that > in the doc (hopefully, it's an company internal code so they should be > trained), it will be a bummer to debug. > > I still don't get it... If you have a framework you presumably have an > entry point. Why can't you set up your policy in that entrypoint? Why > would a user attempt to change the policy at runtime (you haven't > listed examples of libraries that do this)? I see a lot of "I want to > protect users from ..." arguments but I haven't yet seen "this and > that happened in production and we haven't been able to debug what > happened for a while". > > Do you handle cases when people install a blocking logging handler in > their async application? Do you handle cases when a malfunctioning > sys.excepthook is installed? What about cases when users accidentally > import gevent somewhere in their asyncio application and it > monkeypatches the 'socket' module (this is a real horror story, by the > way)? My point is that there are so many things that users can do > that will break any framework, be it asyncio or django or trio. > > This sounds like "if something can happen it will happen" kind of > thing, but I haven't yet seen good examples of real code that suffers > from non-locked policies. Using the nurseries example doesn't count, > as this is something that we want to have as a builtin functionality > in 3.8. > > Locking policies can lead to more predictable user experience; OTOH > what happens if, say, aiohttp decides to lock its policy to use uvloop > and thus make it impossible for its users to use tokio or some other > loop implementation? > > Yury > > > > On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin > wrote: > > > > I like it. > > > > First, it solves the issue for policies, and let people decide how they > > want to deal with the problem (drop the lib, subclass the > > policy/factory, etc). > > > > But it also solves the problem for loops, because loops are set by the > > task factory, and so you can easily check somebody is changing your loop > > from you locked policy and do whatever you want. > > > > This also solves the problem of: > > > > - task factories > > - event loop life cycle hooks > > > > Indeed, if somebody needs those, he/she can implement a custom loop, > > which can be safe guarded by the policy, which is locked. > > > > It doesn't have the drawback of my proposal of being overly general, and > > is quite simple to implement. But it does let people get creative with > > the stack. > > > > > > > > > > ___ > > Python-ideas mailing list > > Python-ideas@python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > Yury > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thanks, Andrew Svetlov ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
> I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug. I still don't get it... If you have a framework you presumably have an entry point. Why can't you set up your policy in that entrypoint? Why would a user attempt to change the policy at runtime (you haven't listed examples of libraries that do this)? I see a lot of "I want to protect users from ..." arguments but I haven't yet seen "this and that happened in production and we haven't been able to debug what happened for a while". Do you handle cases when people install a blocking logging handler in their async application? Do you handle cases when a malfunctioning sys.excepthook is installed? What about cases when users accidentally import gevent somewhere in their asyncio application and it monkeypatches the 'socket' module (this is a real horror story, by the way)? My point is that there are so many things that users can do that will break any framework, be it asyncio or django or trio. This sounds like "if something can happen it will happen" kind of thing, but I haven't yet seen good examples of real code that suffers from non-locked policies. Using the nurseries example doesn't count, as this is something that we want to have as a builtin functionality in 3.8. Locking policies can lead to more predictable user experience; OTOH what happens if, say, aiohttp decides to lock its policy to use uvloop and thus make it impossible for its users to use tokio or some other loop implementation? Yury On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin wrote: > > I like it. > > First, it solves the issue for policies, and let people decide how they > want to deal with the problem (drop the lib, subclass the > policy/factory, etc). > > But it also solves the problem for loops, because loops are set by the > task factory, and so you can easily check somebody is changing your loop > from you locked policy and do whatever you want. > > This also solves the problem of: > > - task factories > - event loop life cycle hooks > > Indeed, if somebody needs those, he/she can implement a custom loop, > which can be safe guarded by the policy, which is locked. > > It doesn't have the drawback of my proposal of being overly general, and > is quite simple to implement. But it does let people get creative with > the stack. > > > > > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Yury ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
I like it. First, it solves the issue for policies, and let people decide how they want to deal with the problem (drop the lib, subclass the policy/factory, etc). But it also solves the problem for loops, because loops are set by the task factory, and so you can easily check somebody is changing your loop from you locked policy and do whatever you want. This also solves the problem of: - task factories - event loop life cycle hooks Indeed, if somebody needs those, he/she can implement a custom loop, which can be safe guarded by the policy, which is locked. It doesn't have the drawback of my proposal of being overly general, and is quite simple to implement. But it does let people get creative with the stack. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
Policy locking is a viable idea at first glance. On Sun, Jun 10, 2018 at 8:05 AM Nick Coghlan wrote: > On 10 June 2018 at 07:59, Michel Desmoulin > wrote: > >> What I'm proposing is to make that easy to implement by just letting >> anyone put a check in there. >> >> Overriding policy, loops or tasks factories are usually down for >> critical parts of the system. The errors emerging from a bug in there >> are very cryptic. >> >> Asyncio design made the choice to expose very low level things. You >> literally don't have this problem in languages like JS because nobody >> can change those. >> >> Now it's here, it's a footgun, and it would be nice to provide a way to >> put it in a holster. >> > > With the API need framed that way, perhaps all that asyncio is currently > missing is an "asyncio.lock_policy(unlock_token, err_callback)" API such > that your application can declare that initialisation is completed and no > further event loop policy changes should be allowed? > > (The "unlock_token" would be an arbitrary app-provided object that must > also be passed to the corresponding "unlock_policy" call - that way > libraries couldn't unlock the policy after the application locks it, since > they won't have a reference to the app-specific unlock token). > > Adding further callback hooks for more events seems like it will just push > the problem back another level, and you'll have the potential for conflicts > between callbacks registered with the new hooks, and an even harder to > understand overall system. > > By contrast, the above would be amenable to doing something like: > > 1. Per-process setup code establishes a particular event loop policy, > and then locks it > 2. Until the policy gets unlocked again, attempts to change it will > call the err_callback (so the app can raise a custom access denied > exception) > 3. get_event_loop(), set_event_loop(), and new_event_loop() are all > already managed by the event loop policy, so shouldn't need new hooks > 4. stop(), close(), set_debug(), set_task_factory(), etc are all > already managed by the event loop (and hence by the event loop policy), so > shouldn't need new hooks > > Right now, the weak link in that chain is that there's no way for the > application to keep a library from switching out the event policy with a > new one, and subsequently bypassing all of the app's control over how it > expects event loops to be managed. Given a way to close that loophole, the > application should already have the ability to enforce everything else that > it wants to enforce via the existing event loop and event loop policy APIs. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thanks, Andrew Svetlov ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
On 10 June 2018 at 07:59, Michel Desmoulin wrote: > What I'm proposing is to make that easy to implement by just letting > anyone put a check in there. > > Overriding policy, loops or tasks factories are usually down for > critical parts of the system. The errors emerging from a bug in there > are very cryptic. > > Asyncio design made the choice to expose very low level things. You > literally don't have this problem in languages like JS because nobody > can change those. > > Now it's here, it's a footgun, and it would be nice to provide a way to > put it in a holster. > With the API need framed that way, perhaps all that asyncio is currently missing is an "asyncio.lock_policy(unlock_token, err_callback)" API such that your application can declare that initialisation is completed and no further event loop policy changes should be allowed? (The "unlock_token" would be an arbitrary app-provided object that must also be passed to the corresponding "unlock_policy" call - that way libraries couldn't unlock the policy after the application locks it, since they won't have a reference to the app-specific unlock token). Adding further callback hooks for more events seems like it will just push the problem back another level, and you'll have the potential for conflicts between callbacks registered with the new hooks, and an even harder to understand overall system. By contrast, the above would be amenable to doing something like: 1. Per-process setup code establishes a particular event loop policy, and then locks it 2. Until the policy gets unlocked again, attempts to change it will call the err_callback (so the app can raise a custom access denied exception) 3. get_event_loop(), set_event_loop(), and new_event_loop() are all already managed by the event loop policy, so shouldn't need new hooks 4. stop(), close(), set_debug(), set_task_factory(), etc are all already managed by the event loop (and hence by the event loop policy), so shouldn't need new hooks Right now, the weak link in that chain is that there's no way for the application to keep a library from switching out the event policy with a new one, and subsequently bypassing all of the app's control over how it expects event loops to be managed. Given a way to close that loophole, the application should already have the ability to enforce everything else that it wants to enforce via the existing event loop and event loop policy APIs. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
>
> IMHO, it is not any framework's job to check for this. It is a
> programmer error.
Not clearing the memory is a programmer error either but a gc helps.
Not closing a file either but using `with` help.
Not passing the proper type is a programmer error but we have type hints.
We even have assert that we can be disabled to check that the arguments
of a function match a test when debugging.
You don't need to babysit programmers so much.
Helping to debug is not babysitting. It's a important part of the user
experience, and it's especially needed on asyncio, one of the hardest
part of the stdlib to code with.
Not if the cost is adding new APIs.
The cost of bugs is greater than the cost of API. The current API gives
no sane way to prevent the bug. You can't expect anyone to check if the
policy / loop / task factory has changed every single time in the code
you depend on it.
>
> I agree with Andrew, if we open this precedent, next thing we know,
> Python has to provide callbacks for any internal state changes.
It's not internal, it's a public API.
Why not
> callbacks for when modules are imported?
Good example.
We have import hooks to run code every time a module is imported :
https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-hooks
Etc. etc. It leads to API
> noise.
Every features is either useful or noise. I argue that this one is useful.
Doesn't seem to be justified in this case, since I would guess
> almost all applications only change event loop policy once, during
> startup, and never again.
Yes, but when is start up ?
E.G: I'm currently working on a large QT api on an old 2.7 code base. It
has threads to avoid blocking the UI, qt events loops of course and the
tornado events loop providing a web API. The orchestration of that very
complex app requires careful initialization for a start up time of about
7s, and we had to code some wrapper and document it as the only entry
point to make sure that if a junior breaks things the error message is
very clear.
RuntimeError('The event loop should not be set when...') is way easier
to debug than an obscure down the road error on some data structure that
is incorrectly accessed.
What I'm proposing is to make that easy to implement by just letting
anyone put a check in there.
Overriding policy, loops or tasks factories are usually down for
critical parts of the system. The errors emerging from a bug in there
are very cryptic.
Asyncio design made the choice to expose very low level things. You
literally don't have this problem in languages like JS because nobody
can change those.
Now it's here, it's a footgun, and it would be nice to provide a way to
put it in a holster.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
On Sat, 9 Jun 2018 at 14:31, Michel Desmoulin
wrote:
>
>
> Le 09/06/2018 à 12:33, Andrew Svetlov a écrit :
> > If we consistently apply the idea of hook for internal python structure
> > modification too many things should be changed. Import
> > machinery, tracemalloc, profilers/tracers, name it.
> > If your code (I still don't see the real-life example) wants to check a
> > policy change -- just do it.
> >
> > # on initialization
> > policy = asyncio.get_event_loop_policy()
> >
> > # somewhere in code
> > if policy is not asyncio.get_event_loop_policy():
> > raise RuntimeError("Policy was changed")
>
> You need to repeat this check everywhere because nothing guaranty a code
> hasn't change it after the check.
>
> While a call back allow you to catch it every time.
>
> You can set either raise, warn, monkey patch, disable features, etc.
>
IMHO, it is not any framework's job to check for this. It is a programmer
error. You don't need to babysit programmers so much. Not if the cost is
adding new APIs.
I agree with Andrew, if we open this precedent, next thing we know, Python
has to provide callbacks for any internal state changes. Why not callbacks
for when modules are imported? Etc. etc. It leads to API noise. Doesn't
seem to be justified in this case, since I would guess almost all
applications only change event loop policy once, during startup, and never
again.
--
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
Le 09/06/2018 à 12:33, Andrew Svetlov a écrit :
> If we consistently apply the idea of hook for internal python structure
> modification too many things should be changed. Import
> machinery, tracemalloc, profilers/tracers, name it.
> If your code (I still don't see the real-life example) wants to check a
> policy change -- just do it.
>
> # on initialization
> policy = asyncio.get_event_loop_policy()
>
> # somewhere in code
> if policy is not asyncio.get_event_loop_policy():
> raise RuntimeError("Policy was changed")
You need to repeat this check everywhere because nothing guaranty a code
hasn't change it after the check.
While a call back allow you to catch it every time.
You can set either raise, warn, monkey patch, disable features, etc.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
Hold on. aiohttp doesn't suffer from hooks absence. Moreover, I don't see how these hooks could be utilized by aiohttp. Gunicorn workers are not imported and instantiated by user code, they are imported by gunicorn using a command line parameter. Please choose a different use case as the proof of your request. On Wed, Jun 6, 2018 at 1:41 PM Michel Desmoulin wrote: > > Hi Yuri, > > > > > I actually want to propose to reduce policies API surface by > > deprecating and then removing "set_child_watcher()" methods. Besides, > > there are many other higher priority To-Do items for asyncio in 3.8, > > like implementing Trio's nursery-like objects and cancellation scopes > > or fixing tracebacks in Tasks. > > > > That said, the above is my "IMO". And in your email you haven't > > actually provided clear scenarios that could be solved by adding > > "event loop hooks" to asyncio. So I have a few questions for you: > > > > - Do you have real-life examples of libraries that abuse policies in > > some weird ways? > > It's not abuse, it's just regular use of a documented feature, but a > very potent feature with deep consequences. > > > - Are those libraries popular? > > aiothttp comes to mind. But the fact is, I think people explicitly avoid > using policies and custom loops because they know that they can be > swapped and you can't make it a hard requirement since they can be > swapped under your node. > > > - What's the actual problem they try to solve by using policies? > > e.g: > > https://github.com/aio-libs/aiohttp/blob/53828229a13dc72137f430abc7a0c469678f3dd6/aiohttp/worker.py > > > - What problem are you trying to solve in your code that uses policies? > > I have tried to create an abstraction that creates an uvloop if it > exists, or a regular loop otherwise, or integrated it to the twisted > reactor or the trio event loop if it's loaded. > > I want to abstract that from the user, so I tried to put that in a > policy. But that's dangerous since it can be changed at any time, so I > gave up on it and made it explicit. Of course, if the user misses that > in the doc (hopefully, it's an company internal code so they should be > trained), it will be a bummer to debug. > > Another example is the proof of concept of nurseries we talked about on > twitter: > > > https://0bin.net/paste/V5KyhAg-2i5EOyoK#dzBvhdCVeFy8Q2xNcxXyqwtyQFgkxlKI3u5QG0buIcT > > Yet another one, with a custom loop this time: > > I want to provide a fail fast mode for code using loop.run_forever. The > goal is to make it crashes when an uncaught exception occurs instead of > just log it in the console to ease debugging in a local machine. > > Of course it would be best to use run_until_complete instead but you > don't always get to choose. > > So we set a task factory on the loop. But, of course, you lose > everything if something changes the loop on the fly, which my code has > no idea has happened. > > Another use case is to just log that the loop / policy has changed in > debug mode. It's always something I want to know anyway, because it has > consequences on my entire program since those are pretty fundamental > components. > > > - Why do you think this isn't a documentation/tutorial issue?> - Can you > list 2-3 clear examples where having hooks would benefit an > > average asyncio user? > > The best documentation is the one you don't need to write. But if I'm > being honest, I'd like to have it despite having a good warning in the > documentation. > > When you run Django's manage.py runserver, it will check your code for a > lot of common issues and raise a warning or an exception, letting you > know what to do. > > With those hooks, I could check if a policy or a loop is changed, and if > some things in my code depends on a policy or a loop, I can do the same > and raise a warning or an exception. > > This benefit the users because they get the info exactly in context, and > not just rely on them reading the doc, understanding it, remembering it > and applying it. > > And that benefits the framework writers because that's less support, and > less bug reports to deal with. > > Using asyncio is scary and mysterious enough for a lot of people, so I > want to make the experience as natural as possible. I don't want people > to have to read my doc and learn what event loops and policies are for > basic usage. It's too much. But on the other hand, I do want them to be > able to debug a problem if the loop or policy is swapped. > > > > > Thank you, > > Yury > > Thank you too > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thanks, Andrew Svetlov ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
Hi Yuri, > > I actually want to propose to reduce policies API surface by > deprecating and then removing "set_child_watcher()" methods. Besides, > there are many other higher priority To-Do items for asyncio in 3.8, > like implementing Trio's nursery-like objects and cancellation scopes > or fixing tracebacks in Tasks. > > That said, the above is my "IMO". And in your email you haven't > actually provided clear scenarios that could be solved by adding > "event loop hooks" to asyncio. So I have a few questions for you: > > - Do you have real-life examples of libraries that abuse policies in > some weird ways? It's not abuse, it's just regular use of a documented feature, but a very potent feature with deep consequences. > - Are those libraries popular? aiothttp comes to mind. But the fact is, I think people explicitly avoid using policies and custom loops because they know that they can be swapped and you can't make it a hard requirement since they can be swapped under your node. > - What's the actual problem they try to solve by using policies? e.g: https://github.com/aio-libs/aiohttp/blob/53828229a13dc72137f430abc7a0c469678f3dd6/aiohttp/worker.py > - What problem are you trying to solve in your code that uses policies? I have tried to create an abstraction that creates an uvloop if it exists, or a regular loop otherwise, or integrated it to the twisted reactor or the trio event loop if it's loaded. I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug. Another example is the proof of concept of nurseries we talked about on twitter: https://0bin.net/paste/V5KyhAg-2i5EOyoK#dzBvhdCVeFy8Q2xNcxXyqwtyQFgkxlKI3u5QG0buIcT Yet another one, with a custom loop this time: I want to provide a fail fast mode for code using loop.run_forever. The goal is to make it crashes when an uncaught exception occurs instead of just log it in the console to ease debugging in a local machine. Of course it would be best to use run_until_complete instead but you don't always get to choose. So we set a task factory on the loop. But, of course, you lose everything if something changes the loop on the fly, which my code has no idea has happened. Another use case is to just log that the loop / policy has changed in debug mode. It's always something I want to know anyway, because it has consequences on my entire program since those are pretty fundamental components. > - Why do you think this isn't a documentation/tutorial issue?> - Can you list > 2-3 clear examples where having hooks would benefit an > average asyncio user? The best documentation is the one you don't need to write. But if I'm being honest, I'd like to have it despite having a good warning in the documentation. When you run Django's manage.py runserver, it will check your code for a lot of common issues and raise a warning or an exception, letting you know what to do. With those hooks, I could check if a policy or a loop is changed, and if some things in my code depends on a policy or a loop, I can do the same and raise a warning or an exception. This benefit the users because they get the info exactly in context, and not just rely on them reading the doc, understanding it, remembering it and applying it. And that benefits the framework writers because that's less support, and less bug reports to deal with. Using asyncio is scary and mysterious enough for a lot of people, so I want to make the experience as natural as possible. I don't want people to have to read my doc and learn what event loops and policies are for basic usage. It's too much. But on the other hand, I do want them to be able to debug a problem if the loop or policy is swapped. > > Thank you, > Yury Thank you too ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
Twisted's reactor API has some lifecycle hooks: https://twistedmatrix.com/documents/18.4.0/api/twisted.internet.interfaces.IReactorCore.html#addSystemEventTrigger My impression is that this is actually pretty awkward for twisted/asyncio interoperability, because if you're trying to use a twisted library on top of an asyncio loop then there's no reliable way to implement these methods. And twisted uses these internally for things like managing its thread pool. There is some subtlety though because in twisted, reactors can't transition from the stopped state back into the running state, which implies an invariant where the start and shutdown hooks can be called at most once. Anyway, I'm not a twisted expert, but wanted to flag this so you all know that if you're talking about adding lifecycle hooks then you know to go talk to them and get the details. (Trio does have some sorta-kinda analogous functionality. Specifically it has a concept of "system tasks" that are automatically cancelled when the main task exits, so they have a chance to do any cleanup at that point. But trio's lifecycle model is so different that I'm not sure how helpful this is.) -n On Tue, Jun 5, 2018, 05:48 Michel Desmoulin wrote: > After years of playing with asyncio, I'm still having a harder time > using it than any other async architecture around. There are a lot of > different reasons for it, but this mail want to address one particular one: > > The event loop and policy can be tweaked at any time, by anyone. > > Now, it's hard enough to have to deal, manually, with a low-level event > loop. But having it exposed that much, and it being that flexible means > any code can just do whatever it wants with it, and make a mess. > > Several things in particular, comes to mind: > > - Changing the event loop policy > - Changing the event loop > - Spawning a new loop > - Starting the loop > - Stopping the loop > - Closing the loop > > Now, if you want to make any serious project with it, you currently have > to guard against all of those, especially if you want to have proper > cleanup code, good error message and a decent debugging experience. > > I tried to do it for one year, and currently, it's very hard. You have a > lot of checks to make, redundantly in a lot of places. Some things can > only be done by providing a custom event policy/loop yourself, and, of > course, expecting (aka documenting and praying) that it's used. > > For a lot of things, when it breaks, the people that haven't read the > doc in depth will have a hard time to understand the problem after the > fact. > > Sometimes, it's just that your code use somebody else code that is not > here to read your doc anymore. Now you have to check their code to > understand what they are doing that breaks your expectations about the > loop / policy or workflow. > > Barring the creating of an entire higher level framework that everybody > will agree on using and that makes messing up way harder, we can improve > this situation by adding hooks to those events. > > I hence propose to add: > > - asyncio.on_change_policy(cb:Callable[[EventLoopPolicy, > EventLoopPolicy], EventLoopPolicy]) > > - asyncio.on_set_event_loop(cb:Callable[[EventLoop, EventLoop], EventLoop]) > > - asyncio.on_create_event_loop(cb:Callable[[EventLoop], EventLoop]) > > - EventLoop.on_start(cb:Callable[EventLoop]) > > - EventLoop.on_stop(cb:Awaitable[EventLoop]) > > - EventLoop.on_close(cb:Callable[EventLoop]) > > - EventLoop.on_set_debug_mode(cb:Callable[[loop]]) > > This would allow to implement safer, more robust and easier to debug > code. E.G: > > - you can raise a warning stating that if somebody changes the event > policy, it must inherit from your custom one or deal with disabled features > > - you can raise an exception on loop swap and forbid it, saying that > your small script doesn't support it yet so that it's easy to understand > the limit of your code > > - you can hook on the event loop life cycle to automatically get on > board, or run clean up code, starting logging, warn that you were > supposed to start the loop yourself, etc > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add hooks to asyncio lifecycle
Hi Michel, Yes, theoretically, it's possible to try to change an event loop policy while an event loop is running, but I've yet to see a library (or user code) that tries to do that (it's pointless anyways). There are libraries like uvloop that tell their users to explicitly install a special policy *before* they run their code, but that's about it. The only use case for policies is allowing to plug in a custom event loop implementation. The current policies implementation, as well as "get_event_loop()" and "get_running_loop()" functions, is already very complex. For example, I tried to add rudimentary support for "os.fork()" in 3.7 and failed, because there are too many things that can go wrong. Therefore without a *very* clear case for adding hooks, I'm -1 on further extending and complicating policies API (or any related APIs). I actually want to propose to reduce policies API surface by deprecating and then removing "set_child_watcher()" methods. Besides, there are many other higher priority To-Do items for asyncio in 3.8, like implementing Trio's nursery-like objects and cancellation scopes or fixing tracebacks in Tasks. That said, the above is my "IMO". And in your email you haven't actually provided clear scenarios that could be solved by adding "event loop hooks" to asyncio. So I have a few questions for you: - Do you have real-life examples of libraries that abuse policies in some weird ways? - Are those libraries popular? - What's the actual problem they try to solve by using policies? - What problem are you trying to solve in your code that uses policies? - Why do you think this isn't a documentation/tutorial issue? - Can you list 2-3 clear examples where having hooks would benefit an average asyncio user? Thank you, Yury On Tue, Jun 5, 2018 at 8:48 AM Michel Desmoulin wrote: > > After years of playing with asyncio, I'm still having a harder time > using it than any other async architecture around. There are a lot of > different reasons for it, but this mail want to address one particular one: > > The event loop and policy can be tweaked at any time, by anyone. > > Now, it's hard enough to have to deal, manually, with a low-level event > loop. But having it exposed that much, and it being that flexible means > any code can just do whatever it wants with it, and make a mess. > > Several things in particular, comes to mind: > > - Changing the event loop policy > - Changing the event loop > - Spawning a new loop > - Starting the loop > - Stopping the loop > - Closing the loop > > Now, if you want to make any serious project with it, you currently have > to guard against all of those, especially if you want to have proper > cleanup code, good error message and a decent debugging experience. > > I tried to do it for one year, and currently, it's very hard. You have a > lot of checks to make, redundantly in a lot of places. Some things can > only be done by providing a custom event policy/loop yourself, and, of > course, expecting (aka documenting and praying) that it's used. > > For a lot of things, when it breaks, the people that haven't read the > doc in depth will have a hard time to understand the problem after the fact. > > Sometimes, it's just that your code use somebody else code that is not > here to read your doc anymore. Now you have to check their code to > understand what they are doing that breaks your expectations about the > loop / policy or workflow. > > Barring the creating of an entire higher level framework that everybody > will agree on using and that makes messing up way harder, we can improve > this situation by adding hooks to those events. > > I hence propose to add: > > - asyncio.on_change_policy(cb:Callable[[EventLoopPolicy, > EventLoopPolicy], EventLoopPolicy]) > > - asyncio.on_set_event_loop(cb:Callable[[EventLoop, EventLoop], EventLoop]) > > - asyncio.on_create_event_loop(cb:Callable[[EventLoop], EventLoop]) > > - EventLoop.on_start(cb:Callable[EventLoop]) > > - EventLoop.on_stop(cb:Awaitable[EventLoop]) > > - EventLoop.on_close(cb:Callable[EventLoop]) > > - EventLoop.on_set_debug_mode(cb:Callable[[loop]]) > > This would allow to implement safer, more robust and easier to debug > code. E.G: > > - you can raise a warning stating that if somebody changes the event > policy, it must inherit from your custom one or deal with disabled features > > - you can raise an exception on loop swap and forbid it, saying that > your small script doesn't support it yet so that it's easy to understand > the limit of your code > > - you can hook on the event loop life cycle to automatically get on > board, or run clean up code, starting logging, warn that you were > supposed to start the loop yourself, etc > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Yury ___
