On Tue, 2007-11-13 at 17:08 -0500, Alan Conway wrote:
> Andrew Stitcher wrote:
> > On Tue, 2007-11-13 at 14:44 -0500, Alan Conway wrote:
> >> Andrew, does the IO framework support non-IO wakeup of threads yet? I
> >> think that's the key to eliminating the serializer and
> >> getting message delivery into the thread pool. I'd like to be able to
> >> mark a queue so that it can wake an IO thread when it's got messages.
> >
> > I don't think that anything you describe below needs non-IO wakeup of
> > the threads! Although time based wakeup might turn out to be useful I
> > have no plans to implement it the moment.
> Not at all interested in time based wakeup. What I describe below is
> wakeup based on presence of messages on a queue. How do we do that with
> only IO based wakeup?
As I say below this equates to saying we would like to be notified of
writeability of all connections that are subscribed to the queue. So
that we can then get a callback in the context of the writing thread for
that connection.
As you infer below, if a connection is writeable but there is nothing
presently available to write there would have been a callback notifying
the writeability, if subsequently there was nothing queued for writing
then the connection would no longer be monitoring for writeability (to
do otherwise would impose a high overhead of useless callbacks). As soon
as there is something to write then writeability needs to be monitored
again.
>
> >> I think we can drive message delivery from IO wakeups like this:
> >>
> >> Producer ready-to-read wakeup: only goes as far as enqueueing the message.
> >
> > this is already how it works.
> Currently the producer calls Queue::deliver which calls the Serializer.
> I think we agree that we'd like to get rid of the serializer call at
> this point.
Sorry I was probably thinking more of the previous scheme.
>
> >> Consumer thread ready-to-write wakeup: visits queues that it consumes
> >> from and
> >> - if it finds messages, write until writing would block, then go back
> >> to sleep.
> >> - if there's not enough data to block writing, mark myself as an
> >> "unsatisfied consumer" on all queues of interest.
> >> - Any queue with unsatisfied consumers enables IO wakeups.
> >
> > Yes this would be a good way to make it work (I think this the general
> > scheme that Gordon and I have been pushing for a while now). The IO
> > framework already has a place to hook this into (via a yet-as-unwritten
> > callback).
>
> Great. So do you or Gordon have plans to implement that in the short
> term? If not I can look into it.
I'd like to do this when I'm done with the low level stuff, but that
doesn't seem very short term.
>
> >> Queue-has-messages wakeup:
> >> - write messages to unsatisfied consumers, writing until write would
> >> block.
> >> - remove unsatisfied consumer marker for all consumers that are blocked
> >> in write, leave it for any that are not.
> >
> > What this needs to do is poke the IO framework to tell it that we are
> > now interested in writing on a specific connection.
>
> Potentially a set of connections. There may be more than one consumer
> interested in the queue.
Ack
>
> so that if we
> > weren't looking for writeability before we will now. This is already
> > possible.
>
> So are you saying that if a consumer is writable but there are no
> messages to write it would mark itself "unsatisfied" on the queues it is
> interested in and disable it's write trigger (or rather, not re-enable
> its write trigger). When a queue has messages again it would enable the
> write triggers of unsatified consumers causing their write threads to
> wake up and contend for messages to write. That avoids the need for
> non-IO wakeup. Is that what you mean? (presumably epoll is level
> triggered for this to work)
There is no explicit "unsatisfied" mark, it just comes naturally from
informing the connections you have something to write to them, and
writing it in response to a "we're writable, but have nothing queued"
callback.
>
> >
> >> The idea is that every consumer is always in exactly one of these states:
> >> - blocked on write to network.
> >> - blocked waiting for message from a set of queues.
> >> - being serviced by a thread (which could be triggered by
> >> ready-to-write or a queue-has-messages, but not both)
Except that there's no "blocking" as such ("waiting" might be a better
word), and these states are summaries of implicit state in the queue and
asynch IO state of the connection.
Actually I'm not sure what you mean by "consumer" in this context. Is it
the connection, the queue or some combination of the two.
So to recast these states:
In queue:
1. No messages queued
get callback informing connection writable:
do nothing (there's nothing to write)
goto state 1
receive messages to queue:
set connections to monitor writability
goto state 2
2. Queued messages
get callback informing connection writable:
queue all messages for write
goto state 1
I think that's all you need here.
Is what I'm talking about becoming clear?
Andrew
