[issue30844] selectors: Add urgent data to read event
Pim Klanke <p...@protonic.nl> added the comment: On 02-11-17 16:54, STINNER Victor wrote: > STINNER Victor <victor.stin...@gmail.com> added the comment: > > It seems like kqueue supports urgent data: > > "EV_OOBAND: Read filter on socket may set this flag to indicate the presence > of out of band data on the descriptor." > > Example: > https://github.com/daurnimator/cqueues/commit/52baaf1c25bc7e6f7cb4685cb05f4ed47a3f404a Looks promising. I will have to look into this. > Be careful, I also found: > > // Older versions of Mac OS X may not define EV_OOBAND. > #if !defined(EV_OOBAND) > # define EV_OOBAND EV_FLAG1 > #endif // !defined(EV_OOBAND) > > -- > > ___ > Python tracker <rep...@bugs.python.org> > <https://bugs.python.org/issue30844> > ___ > -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Pim Klanke <p...@protonic.nl> added the comment: On 26-09-17 14:13, STINNER Victor wrote: > > STINNER Victor added the comment: > > It would help to look how Twisted, eventlet, gevent and others handle "urgent > data" and "exceptions". First of all, there are no exceptions, only "exceptional conditions". Second of all, There is no "urgent data" AND "exceptional conditions"; "urgent data" is one of possible exceptional conditions, but not all exceptional conditions mean there is urgent data to read. Check if they succeeded to formalize these events. IMO, they have made it more confusing by using the word 'exception'. In this case I would have stuck with 'exceptional'. But at least it is better than having to replace it with 'urgent data' > > asyncore uses select() or poll(). > > asyncore.poll() uses select.select(). It adds the fd to exceptfds if the fd > is in the readfds or writefds, then asyncore calls _exception(): > > r = []; w = []; e = [] > for fd, obj in map.items(): > is_r = obj.readable() > is_w = obj.writable() > (...) > if is_r or is_w: > e.append(fd) > (...) > > try: > r, w, e = select.select(r, w, e, timeout) > except select.error, err: > (...) > > (...) > > for fd in e: > obj = map.get(fd) > if obj is None: > continue > _exception(obj) > > asyncore.poll2() uses select.poll(). It only uses POLLPRI if the fd is > readable but always checks for error condition (POLLERR) (if asyncio waits > for read and/or write events): It registers for POLLERR, POLLHUP and POLLNVAL, events one does need not to subscribe for. These events are output events only and should not be added to mask. > > for fd, obj in map.items(): > flags = 0 > if obj.readable(): > flags |= select.POLLIN | select.POLLPRI > # accepting sockets should not be writable > if obj.writable() and not obj.accepting: > flags |= select.POLLOUT > if flags: > # Only check for exceptions if object was either readable > # or writable. > flags |= select.POLLERR | select.POLLHUP | select.POLLNVAL > pollster.register(fd, flags) > Adding POLLPRI to the flags if fd is readable doesn't cut it when using it with sysfs gpio kernel driver. The fd for a sysfs input is ALWAYS readable. We want to ONLY wake on POLLPRI event. > -- > > ___ > Python tracker <rep...@bugs.python.org> > <https://bugs.python.org/issue30844> > ___ > -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Pim Klanke <p...@protonic.nl> added the comment: On 26-09-17 14:01, STINNER Victor wrote: > > STINNER Victor added the comment: > > Anothe piece of history, the creation of the selectors module, bpo-16853, > directly with the win32 "select.select(r, w, w, timeout)": > > commit 243d8d85debaa319a2be0143003a9e881a0f5646 > Author: Charles-François Natali <cf.nat...@gmail.com> > Date: Wed Sep 4 19:02:49 2013 +0200 > > Issue #16853: Add new selectors module. > > -- 8 or so weeks ago I've send an email to Charles-François Natali (a.k.a. neologix). I asked him if he could explain the wrapper method. Neologix is in the nosy list as well, but has not answered any questions up till now > > ___ > Python tracker <rep...@bugs.python.org> > <https://bugs.python.org/issue30844> > ___ > -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Pim Klanke <p...@protonic.nl> added the comment:
On 26-09-17 14:10, STINNER Victor wrote:
>
> STINNER Victor added the comment:
>
> On Windows, exceptfds of select() is not only related to urgent ("out of
> band") data, it also notifies connect() failure:
>
> exceptfds:
>
> If processing a connect call (nonblocking), connection attempt failed.
> OOB data is available for reading (only if SO_OOBINLINE is disabled).
>
> https://msdn.microsoft.com/en-us/library/windows/desktop/ms740141(v=vs.85).aspx
Ok. So what is achieved with the wrapper method is to catch a connection
attempt failure error on the writefds, by signaling a WRITE_EVENT on the
writefd.
Application will then most likely try to write to the fd and an error
will occur, telling us the connection attempt failed.
I will try to change the wrapper method so that it still contains this
hack (because that is what it is IMO), but also allows other fds to be
monitored for exceptional events.
>
> I'm not sure that we can easily simplify the third parameter of select() as
> "urgent data". The exact semantics seems to not be portable at all.
The goal of the patch is not to simplify the third parameter of select,
but simply to extend the selectors module with the ability to subscribe
for the POLLPRI ('urgent data to read') event, so selectors module can
be used with sysfs gpio kernel driver.
To support this in SelectSelector the third parameter of select needs to
be used, causing other exceptional events like POLLERR and POLLHUP to
trigger select as well. For PollLikeSelector, this has always been so,
because one does not need to register for POLLERR and POLLHUP (and
POLLNVAL).
The goal of the selectors module is to offer a platform-independent
abstraction layer on top of I/O monitoring functions in select. Our goal
is to extend this with the ability to handle exceptional conditions.
Because some people get confused using the word 'exceptional' when
programming in Python, I was asked to use one of the accepted synonyms.
Since POLLPRI is the only exceptional condition event we explicitly need
to register for, this is the description we choose. IMHO, this
description makes what we try to accomplish harder to understand, than
it was confusing.
>
> poll() describes better the event types. Extract of my local Linux poll()
> manual:
>
> POLLPRI: "There is urgent data to read (e.g., out-of-band data on TCP socket;
> pseudoterminal master in packet mode has seen state change in slave)."
>
> POLLERR: "Error condition (only returned in revents; ignored in events)."
>
> POLLHUP: "Hang up (only returned in revents; ignored in events). Note that
> when reading from a channel such as a pipe or a stream socket, this event
> merely indicates that the peer closed its end of the channel. Subsequent
> reads from the channel will return 0 (end of file) only after all outstanding
> data in the channel has been consumed."
>
> etc.
>
> --
>
> ___
> Python tracker <rep...@bugs.python.org>
> <https://bugs.python.org/issue30844>
> ___
>
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue30844>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Pim Klanke added the comment: On 26-09-17 12:29, STINNER Victor wrote: > > STINNER Victor added the comment: > > Using Git history, I found the following commit which added "r, w, x = > select(r, w, w, timeout)" for Windows in tulip (old name of the asyncio > project, when it was developed outside CPython): > > commit 84124f3d725c9931249d083e78f43fcda91c383a > Author: Richard Oudkerk <shibt...@gmail.com> > Date: Fri Jan 18 13:03:09 2013 + > > Minimal chages to make tests pass on Windows > > https://github.com/python/asyncio/commit/84124f3d725c9931249d083e78f43fcda91c383a Thanks! Although this still does not teach us why the test failed on Windows and what is solved by adding the wrapper method, it does suggest that indeed the runtime behaviour will not be affected by removing the wrapper method. The unit test might. To be honest, I did not run unit tests on Windows. Maybe by doing this, it might just tell us exactly what is accomplished with the wrapper method. > > -- > > ___ > Python tracker <rep...@bugs.python.org> > <https://bugs.python.org/issue30844> > ___ > -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Pim Klanke added the comment: On 26-09-17 11:51, STINNER Victor wrote: > > STINNER Victor added the comment: > > I'm not confortable with the change because of following questions: > > * It seems like your patch changes the SelectSelector behaviour on Windows. > How is a selectors user supposed to upgrade his/her code to get the same > behaviour on Python 3.6 and 3.7? Would it make sense to add a flag to > SelectSelectors get the old behaviour? The wrapper method around the winsock select method causes SelectSelector to behave differently on Windows, which is wrong IMHO. The behaviour should be the same. Why this wrapper exists is unclear to me and since it conflicts with the change I want to make, I asked questions about it 8 weeks ago. Now time has passed and no answers were given, I decided to remove it. Without further documentation I do not know how to modify it otherwise. The other solution would be to add the new feature to all OS's but Windows, but again, since the wrapper method defies logic and is undocumented, I chose to remove it. > > * KqueueSelector doesn't support urgent event. How is a selectors user > suppose to be aware of them? Use a blacklist of selectors which doesn't > support urgent events? This list might evolve in the future, so maybe the > selector should announce which events are supported? KqueueSelector raises a ValueError on registering an invalid event. Accepted events defined in the base class are READ, WRITE and URGENT. Accepted events is overruled in KQueueSelector, allowing only READ and WRITE. > > * This change alone is going to change asyncio behaviour on Windows, no? > Because of the SelectSelector behaviour change on Windows. Reviewing the documentation of winsock select, I can think of no reason why this should change the runtime behaviour of SelectSelector, as well as asyncio. > > -- > > ___ > Python tracker <rep...@bugs.python.org> > <https://bugs.python.org/issue30844> > ___ > -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Pim Klanke added the comment: In the selectors module, the winsock select method is wrapped. The second parameter (writefds) is passed to both writefds and exceptfds and the returned number of fds for write and except are summed and returned in the second element of the tuple. Can anyone explain to me why the winsock select method is wrapped like this? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30847] asyncio: selector_events: add_urgent() for urgent data to read, 3rd argument of select.select()
Pim Klanke added the comment: > "Let's say that we got read event on sockets A and B (in an ordered list from > selectors: A, then B), but B gets urgent data: should we handle B urgent data > before not-urgent A data?" IMO No. The same strategy applies. urgent data events on B have priority over other events on B, but not necessarily over events on A. As long as the urgent data events for a certain file object are handled before other events for that file object, it should be fine. > "Would it be possible to let the developer decide how to prioritize events?" At this point asyncio has no support for prioritizing event handlers. If prioritizing should be necessary, it should IMO be implemented in a way that all event handlers can be prioritized and not just these events. I think we can agree that this falls outside the scope of this patch. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30847> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30847] asyncio: selector_events: add_urgent() for urgent data to read, 3rd argument of select.select()
Pim Klanke added the comment: This is in fact the third catagory of select(), "exceptional conditions", but because some find the term "exceptional" confusing when used in a Python module, we decided to use the term "urgent data", borrowed from the poll(2) man page. (see bpo-30844) An example of an exceptional conditions event is "out of band", another one is edge detection from GPIO Sysfs Interface. The patch for fulfilling this issue can and will be supplied if and when bpo-30844 is merged. 30844 originally also contained the patch to asyncio, but I was asked to split it into two separate patches. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30847> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Pim Klanke added the comment: I'm confused about the wrapper method around winsock select and curious to why this is necessary. I have send an email to neologix to share some light on the subject. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Pim Klanke added the comment: > "The selectors API returns a list of (key, events) tuples. So an application > has to iterate on this list twice?" No. "urgent data" means 'urgent' towards other events for thís key (key being the file object), not towards events for other file objects. AFAIK the returned ready list contains a single tuple for each file object, containing all events for that file object. Most likely urgent data events should be handled before handling other events for a given file object, but IMO there is no need to handle urgent data events of all file objects, before handling other events. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30847] asyncio: selector_events: add_urgent() for urgent data to read, 3rd argument of select.select()
Changes by Pim Klanke <p...@protonic.nl>: -- title: asyncio: selector_events: add_excepter(), 3rd argument of select.select() -> asyncio: selector_events: add_urgent() for urgent data to read, 3rd argument of select.select() ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30847> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add urgent data to read event
Changes by Pim Klanke <p...@protonic.nl>: -- title: selectors: Add exceptional conditions event -> selectors: Add urgent data to read event ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add exceptional conditions event
Changes by Pim Klanke <p...@protonic.nl>: -- title: selectors: Add exceptional urgent data event -> selectors: Add exceptional conditions event ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30847] asyncio: selector_events: add_excepter(), 3rd argument of select.select()
Pim Klanke added the comment: To be able to use GPIO Sysfs Interface with asyncio on our embedded platforms, we require exceptional event support in asyncio. depends on bpo-30844 -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30847> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30847] asyncio: selector_events: add_excepter(), 3rd argument of select.select()
New submission from Pim Klanke: depends on bpo-30844 -- components: asyncio messages: 297652 nosy: pklanke, yselivanov priority: normal severity: normal status: open title: asyncio: selector_events: add_excepter(), 3rd argument of select.select() type: enhancement versions: Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30847> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: Add exceptional urgent data event
Changes by Pim Klanke <p...@protonic.nl>: -- title: selectors: add_excepter(), 3rd argument of select.select() -> selectors: Add exceptional urgent data event ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selectors: add_excepter(), 3rd argument of select.select()
Changes by Pim Klanke <p...@protonic.nl>: -- components: -asyncio title: selectors and asyncio: add_excepter(), 3rd argument of select.select() -> selectors: add_excepter(), 3rd argument of select.select() ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selector_events.py lacks exceptional event support
Changes by Pim Klanke <p...@protonic.nl>: -- pull_requests: +2629 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30844] selector_events.py lacks exceptional event support
New submission from Pim Klanke: To be able to use GPIO Sysfs Interface on our embedded platforms we require exceptional event support. -- components: asyncio messages: 297629 nosy: pklanke, yselivanov priority: normal severity: normal status: open title: selector_events.py lacks exceptional event support type: enhancement versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
