Two usage cases, very desirable in Proton, happen to be mostly trivial on POSIX but mostly non-trivial on Windows: multi-threading and external loops. Proton io and selector classes are POSIX-y in look and feel, but IO completion ports are very different and can mimic the POSIX functionality only so far. The documented restrictions I wrote (PROTON-668) tries to curb expectations for cross platform functionality (but still allow use cases like Dispatch).
This, however, is not a cross platform issue. This particular problem is confined to user space and affects all platforms equally. Presumably the API needs fixing or we agree to take a step backwards. Note that Bozo previously pointed out (Proton mailing list) that the pn_io_t API had threading inconsistencies with pn_io_error() and pn_io_wouldblock(). Perhaps a pn_selectable_t should be passed in instead of a socket parameter, or proton should maintain a map of errors and wouldblock state for sockets until they are pn_close'd (as the Windows code already does for completion port descriptor state). The former would be more consistent with proton overall, the latter would require some user space locking. Another possibility could be to pass in pointers to error/wouldblock state as part of the io call. While I still think this is not a Windows issue, and the documentation is supposed to reflect the Dispatch pattern and not handcuff it, here is more about the pn_io_t implementation: Global state is in struct iocp_t, per socket state is in struct iocpdesc_t (see iocp.h, the OS looks after this stuff in POSIX) It has to set up and tear down the Windows socket system. It has to "know" if the application is using an external loop or pn_selector_select Setup the completion port subsystem (unless external loop) It has to find IOCP state for each passed in socket Manage multiple concurrent io operations per socket (N writes + 1 read) Notify PN_READABLE, PN_WRITABLE, etc changes to the selector (if any) for each call Do an elaborate tear down on pn_io_free (drain completions, force close dangling sockets) Regarding the documentation, I looked at Dispatch, which had been using Proton in a multi-threaded manner for some time with considerable success. The old driver.c (now deprecated) allowed simultaneous threads to do pn_connector_process() <- but no two threads sharing a connector/transport/socket pn_driver_wait_n() combined with pn_connector_next() <- only one thread waiting/choosing at a time pn_driver_wakeup() <- any thread, any time, to unstick things everything else (listen accept connect) considered non-thread safe which provides plenty of parallelism if you have more than one connection. The documentation I wrote tried to say that you could do that much on any platform, but no more (without risking undefined behavior). Things (and the documentation) get further complicated by supporting external loops, which prevents the use of IO completion ports completely for a given pn_io_t and uses a different set of system calls. Perhaps the doc restrictions could be summarized as: One pn_io_t/pn_selector_t, one thread -> no restrictions One pn_io_t/pn_selector_t, multi threads -> limited thread safety (Dispatch) One pn_io_t, no pn_selector_t, external loop, one thread -> no restrictions One pn_io_t, no selector, external loop, multi threads -> ??? multiple pn_io_t: doable, but sockets must stick to one pn_io_t Some difficulties you might not expect: Linux doesn't care if sockets move between selectors, or if one thread is reading on a socket while another is writing to the same one. Simple things like this would have major design and performance implications for Proton on Windows. On Wed, Feb 25, 2015 at 11:57 AM, Rafael Schloming <[email protected]> wrote: > Maybe my head is just thick today, but even staring at the docs a couple > times and reading through what you have below, I can't say I quite > understand what you're going for. What are the actual constraints for the > windows APIs and what is the heavyweight stuff pn_io_t is doing? > > --Rafael > > On Wed, Feb 25, 2015 at 1:02 PM, Cliff Jansen <[email protected]> wrote: > >> A pn_io_t is heavyweight in Windows, because it has an opposite usage >> pattern and moves a lot of kernel stuff into user space compared to >> POSIX. >> >> The quoted documentation was my attempt to capture the Dispatch usage >> pattern, which I assumed would be typical of an application trying to >> spread proton engine use between threads: basically single access to >> pn_selector_select() via a condition variable, and no more than one >> thread working on a given selectable (using proton engine >> encoding/decoding etc., not just io). >> >> In the end, we could just add a zillion locks into the Windows code >> and make it look like it is as thread safe as the POSIX counterpart >> (which has implicit safety when it does in the kernel what Windows is >> doing in user space), but that would defeat using IO completion ports >> at all. The documentation was my attempt of balancing performance >> with sophisticated proton usage on multiple platforms. >> >> Note that there is only one pn_selector_t allowed per pn_io_t (a very >> strong Windows completion port requirement, and sockets are bound to a >> single completion port for life). >> >> On Wed, Feb 25, 2015 at 8:52 AM, Rafael Schloming <[email protected]> >> wrote: >> > On Wed, Feb 25, 2015 at 10:49 AM, Ted Ross <[email protected]> wrote: >> > >> >> Would it be safe to assume that any operations on driver->io are not >> >> thread safe? >> >> >> >> Dispatch is a multi-threaded application. It looks to me as though >> >> io->error is a resource shared across the threads in an unsafe way. >> >> >> > >> > Interesting... so this is what the docs say: >> > >> > /** >> > * A ::pn_io_t manages IO for a group of pn_socket_t handles. A >> > * pn_io_t object may have zero or one pn_selector_t selectors >> > * associated with it (see ::pn_io_selector()). If one is associated, >> > * all the pn_socket_t handles managed by a pn_io_t must use that >> > * pn_selector_t instance. >> > * >> > * The pn_io_t interface is single-threaded. All methods are intended >> > * to be used by one thread at a time, except that multiple threads >> > * may use: >> > * >> > * ::pn_write() >> > * ::pn_send() >> > * ::pn_recv() >> > * ::pn_close() >> > * ::pn_selector_select() >> > * >> > * provided at most one thread is calling ::pn_selector_select() and >> > * the other threads are operating on separate pn_socket_t handles. >> > */ >> > >> > I think this has been somewhat modified by the constraints from the >> windows >> > implementation, and I'm not sure I understand completely what the >> > constraints are there, or entirely what is being described above, but on >> > the posix front, the pn_io_t is little more than just a holder for an >> error >> > slot, and you should have one of these per thread. It shouldn't be a >> > problem to use send/recv/etc from multiple threads though so long as you >> > pass in the pn_io_t from the current thread. >> > >> > --Rafael >>
