Hi everyone,
For a while Kestrel <https://github.com/aspnet/KestrelHttpServer>(an
ASP.NET server built on top of libuv) has been using named pipes on Windows
to accept incoming connections on multiple event loops. It basically works
as described in the processes section of the uv book
<http://nikhilm.github.io/uvbook/index.html>, but with a single process.
After doing some profiling, we noticed that the connections accepted on the
duplicated socket on the secondary threads are using event handles instead
of IOCP for async completion on windows, e.g.:
int uv_tcp_write(uv_loop_t* loop,
uv_write_t* req,
uv_tcp_t* handle,
const uv_buf_t bufs[],
unsigned int nbufs,
uv_write_cb cb) {
int result;
DWORD bytes;
uv_req_init(loop, (uv_req_t*) req);
req->type = UV_WRITE;
req->handle = (uv_stream_t*) handle;
req->cb = cb;
/* Prepare the overlapped structure. */
memset(&(req->u.io.overlapped), 0, sizeof(req->u.io.overlapped));
if (handle->flags & UV_HANDLE_EMULATE_IOCP) {
req->event_handle = CreateEvent(NULL, 0, 0, NULL);
if (!req->event_handle) {
uv_fatal_error(GetLastError(), "CreateEvent");
}
req->u.io.overlapped.hEvent = (HANDLE) ((ULONG_PTR) req->event_handle |
1);
req->wait_handle = INVALID_HANDLE_VALUE;
}
...
if (handle->flags & UV_HANDLE_EMULATE_IOCP &&
!RegisterWaitForSingleObject(&req->wait_handle,
req->event_handle, post_write_completion, (void*) req,
INFINITE, WT_EXECUTEINWAITTHREAD | WT_EXECUTEONLYONCE)) {
SET_REQ_ERROR(req, GetLastError());
uv_insert_pending_req(loop, (uv_req_t*)req);
}
This seems to occur because we accept the socket on both the primary loop
and secondary loop. The first call to uv_tcp_accept associates the socket
with the primary loop's completion port which causes the second call to
CreateIoCompletionPort (via uv_pipe_accept->uv_tcp_import) to fail on the
duplicated socket on the secondary thread. This sequence of events is what
causes the UV_HANDLE_EMULATE_IOCP flag to be set.
static int uv_tcp_set_socket(uv_loop_t* loop,
uv_tcp_t* handle,
SOCKET socket,
int family,
int imported) {
...
/* Associate it with the I/O completion port. */
/* Use uv_handle_t pointer as completion key. */
if (CreateIoCompletionPort((HANDLE)socket,
loop->iocp,
(ULONG_PTR)socket,
0) == NULL) {
if (imported) {
handle->flags |= UV_HANDLE_EMULATE_IOCP;
} else {
return GetLastError();
}
}
We were able to work around this issue on Windows by not using pipes, but
instead accepting connections directly on the secondary loops. To prevent
race conditions, we block inside the listen callback in the primary loop
until the secondary loop finishes accepting the new connection. We still
use Unix domain sockets on non-Windows platforms to support multiple loops.
The PR including this change can be found here:
https://github.com/aspnet/KestrelHttpServer/pull/602/files#diff-5cfac6c0d3b3c415c8884d8696ff11f5R15
This gives us about a 15% performance improvement in our plaintext
benchmark: https://github.com/aspnet/Benchmarks#the-scenarios
<https://github.com/aspnet/Benchmarks#the-scenarios>It also happens to work
around an issue we have with named pipes
<https://github.com/aspnet/KestrelHttpServer/issues/582> on Azure Web Apps,
but it's entirely possible we could find a way to make named pipes work if
need be
I understand this use of libuv is off-label, and that the underlying
implementation could change, but is there any concrete reason we shouldn't
do this on Windows? Is there a better method of supporting multiple event
loops on Windows without emulating IOCP?
Thanks,
Stephen
--
You received this message because you are subscribed to the Google Groups
"libuv" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/libuv.
For more options, visit https://groups.google.com/d/optout.
