On Sep 3, 2012, at 8:13 PM, Parvez Shaikh wrote:
Purpose of doing this is to make only one thread access socket
descriptor.
I have to use multiple threads(thread pool that does some mundane
processing of messages) for utilizing multiple cores as much as
possible(each thread access no shared resource except queuing
message for send); also I don't want multiple threads doing "send"
operation on same socket descriptor - which may require lot of lock-
unlock operations(around send) and in absence of lock-unlock data
interleaving would create trouble (I am using STREAM sockets).
Sound reasonable.
What I would like to do is to have a single thread - an event
reactor thread(that dispatches events) access my socket descriptor
and does read and write.
I would do the same.
For read I am using "bufferevents" for their ease of use. However
for write I can't use "bufferevents" (I don't want my send
operations to be delayed,
I don't think your send operations will be delayed by the buffer
event. I would start by using buffer event for both receiving and
sending messages.
further even with multiple messages in bufferevent leads to only one
invocation of read callback - as per my observations - which is not
suitable for me - I'd like for every message that is buffered to
have read callback called.)
The read callback should loop over the messages in the bufferevent and
invoke your application "message received" event with each message.
There are two approaches for this -
1. write callback on socket descriptor would notify me when
descriptor is "write-ready", when I will pump data until I fail with
EAGAIN(this would mean send buffer full). I will return from write
callback and would pump data when descriptor is "write-ready" again.
2. I would define a separate (custom) event on socket(no EV_READ or
EV_WRITE just EV_PERSIST with no timeout). A thread that enqueues
data for "send" activates the event after enqueuing. An event
callback would access the data that was enqueued and pump it on
descriptor until EAGAIN. I will return from event's callback and
would try pumping data again when data is enqueued/
These are not separate approaches - you must use both if you want to
pump out messages as soon as possible.
First, the socket is ready for writing, and there are no messages in
the queue - write event should not be enabled in this case. The event
loop is waiting for read event.
Then some messages enter the message queue - you must wake up the
event loop to drain your message queue. Depending on the amount of
data and the connection to the other machine, you may get EAGAIN
before the message queue is empty.
If no new message enter the queue, and you don't enable EV_WRITE
event, the rest of messages in the queue may sit there for unlimited
time. So you want to enable EV_WRITE, and the event loop will wake and
continue to drain your messages.
Somehow I am inclined to option 1 (tried option 2 and it worked)
because libevent tells write readiness (when probability of EAGAIN
would be less at least for first write) and in option 2 there is
danger that every time I try to send data I may get EAGAIN for many
messages leading to wasted event activation (rare but possible?).
If you get too many write failures, you can write only when not
waiting for write readiness. You will have to collect stats when
running with real load to see if this is a problem.
However, if you get lot of write failures, it means that you send too
much data and the real problem is how to limit and monitor your
message queue size.
On Mon, Sep 3, 2012 at 7:03 PM, Nir Soffer <nir...@gmail.com> wrote:
On Mon, Sep 3, 2012 at 1:05 PM, Parvez Shaikh
<pshaikh.wo...@gmail.com> wrote:
Thanks Oleg and Nir.
I am also pursuing another approach. Here I have defined a custom
event (associated with FD but not having either EV_READ or EV_WRITE)
and there is no time out either.
A thread queues data to "outgoing message list" and activates an
event. In response to this, event reactor thread (one with dispatch)
calls a function that sends message from this list one by one until
"send" encounters EAGAIN.
Requesting your feedback on this approach.
If you must create your messages in another thread (maybe it is cpu
bound, or you must use blocking operations), this should work. But
if you don't have too, I would not use threads since I don't like
deadlocks :-)
Anyway, when the send fails with EAGAIN, you want to enable
EV_WRITE, so you can continue to drain your messages queue when
socket is ready. The way you work with EV_WRITE is not related to
working with threads.
On Sun, Sep 2, 2012 at 3:45 PM, Nir Soffer <nir...@gmail.com> wrote:
On Sun, Sep 2, 2012 at 6:54 AM, Parvez Shaikh
<pshaikh.wo...@gmail.com> wrote:
Thanks Oleg,
Second approach is what I am doing.
Why disabled EV_WRITE in write callback?
I'd wish to have this callback called again whenever send buffer has
space, so disabling EV_WRITE will prevent this.
You want to enable EV_WRITE only when you have something to write.
When you enable EV_WRITE, libevent will add the descriptor to the
the event backend (e.g select). When the descriptor is ready for
writing, the write callback will be called. If you always enable
EV_WRITE, the callback will be called on every loop iteration, and
the event loop will never wait for events, since the descriptor is
almost always ready for writing.
On Fri, Aug 31, 2012 at 3:54 PM, Oleg <mybrokenb...@gmail.com> wrote:
EV_WRITE calls whenever you can write to send buffer until it's not
full.
So if you have never called send(), but EV_WRITE is enabled you will
receive this event each new loop (your CPU will be ~100% used).
If you have called send() and it didn't return EAGAIN you will also
receive this event next loop.
If you have called send() and it returned EAGAIN you will receive
this event when your send buffer will have some free space.
So for your sending queue it should look like this in pseudo code:
1) Application->Send(data)
{
if (send() != EAGAIN )
return;
else
{
queue.push(data);
enable(EV_WRITE,onWriteCallback);
}
}
2) onWriteCallback()
{
while(!queue.empty)
{
if (send(queue.front())== EAGAIN);
return;
queue.pop();
}
disable(EV_WRITE);
}
31.08.2012, в 8:50, Parvez Shaikh написал(а):
> Hi all,
>
> I have a question about EV_WRITE event as to when does it's
callback function invoked?
>
> Is it that when someone first executes write on an fd associated
with EV_WRITE event?
>
> Or when libevent detects that application can now write to fd
without getting errors?
>
> For EV_READ it is easy to understand that it's callback is invoked
when data is available for read on fd but not clear about EV_WRITE.
>
> Here is what I am trying to do -
>
> I am trying to write asynchronous send/recv application; in which
I will read data on connected sockets asynchronously using
"EV_READ's callback.
>
> For send however, I will enqueue the data to be sent in my own
queue(application will write data to this buffer) and I will flush
the buffer in EV_WRITE callback.
>
> Now if I get the error EAGAIN in send operation in callback of
EV_WRITE, I will simply return and on next invocation of EV_WRITE's
callback I will start flushing my buffer again.
>
> Thanks,
> Parvez
***********************************************************************
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-users in the body.
