[issue30300] asyncio.Controller

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

There doesn't seem to be much appetite for this in the stdlib, so closing.  
It'll just have to live in a third party module.

--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Yury Selivanov]

Yury Selivanov added the comment:

> I appreciate that you want to be careful not to saddle asyncio with too much
> baggage, or that you don't feel Controller is significant enough to generalize
> and put in the package.  Perhaps a middle ground would be to label parts of
> the asyncio API provisional, and add Controller to that?

Thing is, when asyncio was provisional, we still couldn't significantly 
change it or break it.  Never in asyncio stdlib era had we removed or 
redesigned some APIs.  Only small additions and bug fixes.  And honestly,
maintaining something provisional and changing it in bugfix releases
is too much stress: we managed to break `loop.connect_socket` once
because nobody tests bugfix RCs.  It was broken for ~6 months.

IMHO: the design of Controller is currently incomplete (see one of my
previous comments).  Even in this thread two other core devs raised a 
question that the API isn't generic enough to be part of asyncio.
Right now it's not flexible and tailored for one specific use case. 
Should the user need slightly more, they will have to copy/paste it, 
or, worse, inherit from it and use its private APIs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

On May 30, 2017, at 10:36 PM, Yury Selivanov wrote:

>Again, the natural way of something like Controller to end up in asyncio is
>to either go through full PEP process, or live some time on PyPI and prove to
>be useful.

A PEP feels like overkill; we don't require PEPs for every addition to an
existing stdlib module or package.  My worry too is that a PEP tends to evoke
endless bikeshedding.

I appreciate that you want to be careful not to saddle asyncio with too much
baggage, or that you don't feel Controller is significant enough to generalize
and put in the package.  Perhaps a middle ground would be to label parts of
the asyncio API provisional, and add Controller to that?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Yury Selivanov]

Yury Selivanov added the comment:

> There's also value in doing one simple thing that adds convenience for users.
> I don't personally have any interest in building something as elaborate as the
> above, but I've used the simple idea here several times in different projects.

Yeah, I understand.  I totally see how Controller can be useful, moreover,
I've implemented something like it bunch of times (usually for running
tests).  But with Guido no longer actively involved with asyncio, and me
being the only one who is actively working on/supporting asyncio, I 
think we need to go through the full PEP process when we want to add 
new APIs.  bpo simply does not provide enough exposure: guys from
Twisted and Tornado, for instance, won't see this discussion.

I'd be glad to assist you with the PEP though!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Yury Selivanov]

Yury Selivanov added the comment:

> STINNER Victor added the comment:
> 
> Barry: would you be ok to start by adding Controller to asyncio.test_utils,
> and wait later to expose it in the public API?

Sorry, but we are going to deprecate and remove test_utils soon. It's a bunch 
of internal unit test helpers used privately by asyncio. They are not 
documented and not supported. Now that asyncio repo is in CPython repo and we 
don't release it on its own, i see no reason to keep test_utils (we can move it 
to 'Lib/test'.

Again, the natural way of something like Controller to end up in asyncio is to 
either go through full PEP process, or live some time on PyPI and prove to be 
useful.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[STINNER Victor]

STINNER Victor added the comment:

Barry: would you be ok to start by adding Controller to asyncio.test_utils,
and wait later to expose it in the public API?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

On May 29, 2017, at 11:42 PM, Yury Selivanov wrote:

>- detailed logging or hooks to implement it
>- hooks on thread start / stop
>- coroutines to run before starting the server
>- coroutines to run before stopping the loop
>- custom undhandled exceptions handlers
>- type of the server created: TCP/UDP/Unix
>- ability to configure SSL
>- etc
>
>P.S. It would be interesting to try to evolve the idea a bit further: it
>would be interesting if Controller was a high-level description of a service
>and we had a separate concept of ControllerRunner to run Controllers in
>threads, processes and corotuines.  Maybe build something like erlang
>supervisor trees out of them.

There's also value in doing one simple thing that adds convenience for users.
I don't personally have any interest in building something as elaborate as the
above, but I've used the simple idea here several times in different projects.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Yury Selivanov]

Yury Selivanov added the comment:

I'm not sure we want this to be in asyncio: it's a very high-level object 
somewhere in between the low-level and the application level.  Some things off 
the top of my head that users will want from this API:

- detailed logging or hooks to implement it
- hooks on thread start / stop
- coroutines to run before starting the server
- coroutines to run before stopping the loop
- custom undhandled exceptions handlers
- type of the server created: TCP/UDP/Unix
- ability to configure SSL
- etc

Since asyncio is no longer provisional, it won't be possible to evolve the API 
in bugfix releases, which will likely make it impossible to use for many users 
until 3.8.

In general, the advice for things like this is to put them on PyPI, gather some 
feedback, and sort out the API details.

-1 to add this in 3.7 in its current state.

P.S. It would be interesting to try to evolve the idea a bit further: it would 
be interesting if Controller was a high-level description of a service and we 
had a separate concept of ControllerRunner to run Controllers in threads, 
processes and corotuines.  Maybe build something like erlang supervisor trees 
out of them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

On May 29, 2017, at 07:07 AM, Antoine Pitrou wrote:

>For example I might write a UDP server.  Or a distributed system that listens
>to several ports at once, or launches a thread pool. etc.

Thanks, those are nice motivational examples.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Can you elaborate?  What's too specific about it?  Do you have in mind a use 
> case where you wouldn't need to provide hostname and port?

Any use case where setup is more elaborate than calling create_server(...).  
For example I might write a UDP server.  Or a distributed system that listens 
to several ports at once, or launches a thread pool. etc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

Hi Antoine,

On May 28, 2017, at 11:07 AM, Antoine Pitrou wrote:

>I think the API is too specific.

Can you elaborate?  What's too specific about it?  Do you have in mind a use
case where you wouldn't need to provide hostname and port?

>Instead of requiring hostname and port, why not let the user override setup
>and teardown coroutines?
>
>In your case, this could be:
>
>async def setup(self):
>self.server = await self.loop.create_server(...)
>
>async def teardown(self):
>await self.server.wait_closed()

It's certainly possible to factor those out so they could be overridden, I'm
just not sure why that's needed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Antoine Pitrou]

Antoine Pitrou added the comment:

I think the API is too specific.  Instead of requiring hostname and port, why 
not let the user override setup and teardown coroutines?

In your case, this could be:

async def setup(self):
self.server = await self.loop.create_server(...)

async def teardown(self):
await self.server.wait_closed()

--
nosy: +giampaolo.rodola, pitrou, yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

On May 11, 2017, at 12:09 AM, STINNER Victor wrote:

>Why not starting by putting this class in a library to mature its API?

It's already part of aiosmtpd although not with the small amount of
generic-ness included here.  It's been useful and stable.  So this is just
refactoring out the aiosmtpd-ness of it.  It doesn't seem big enough to put
into yet another separate library.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[STINNER Victor]

STINNER Victor added the comment:

I'm not sure that Controller is generic enough to be part of asyncio.

I'm not sure about the cancellation of all pending tasks on stop().

Why not starting by putting this class in a library to mature its API?

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

On May 08, 2017, at 11:06 PM, Nathaniel Smith wrote:

>Looks interesting! What's the advantage over running the server and the test
>in the same loop? The ability to use blocking operations in the tests, and to
>re-use an expensive-to-start server over multiple tests?

So, the ability to re-use expensive-to-start servers is definitely one of the
advantages.  I use nose2's layers, but test fixtures would fall into the same
category.

As for running the server and tests in the same loop; I haven't tried that,
but it seems like it would be more complicated to set up (maybe that's
dependent on the code under test).  More important is that I want to block the
tests until the server starts up.  I'm not sure (haven't tried) whether that's
possible when running them all in the same loop.

One other use case I have is for the LMTP server in Mailman 3.  The controller
turns out to be useful based on the start/stop framework for MM3 "runners".
That's probably strictly doable without the controller, but it's convenient,
readable, and a nice reuse.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Nathaniel Smith]

Nathaniel Smith added the comment:

Looks interesting! What's the advantage over running the server and the test in 
the same loop? The ability to use blocking operations in the tests, and to 
re-use an expensive-to-start server over multiple tests? (I've mostly used 
threads in tests to run blocking code for interoperability testing, and kept 
the async code in the main thread, so this is a bit novel to me.)

--
nosy: +njs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Barry A. Warsaw]

Changes by Barry A. Warsaw :


--
pull_requests: +1594

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30300] asyncio.Controller

[Barry A. Warsaw]

New submission from Barry A. Warsaw:

Over in https://github.com/aio-libs/aiosmtpd we have a Controller class which 
is very handy for testing and other cases.  I realized that this isn't really 
aiosmtpd specific, and with just a few tweaks it could be appropriate for the 
stdlib.

I have a branch ready for a pull request.  This is the tracking/discussion 
issue.

--
components: Library (Lib)
messages: 293205
nosy: barry
priority: normal
severity: normal
status: open
title: asyncio.Controller
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com