Andrew Stitcher wrote:
On Tue, 2007-11-13 at 14:44 -0500, Alan Conway wrote:
Andrew, does the IO framework support non-IO wakeup of threads yet? I
think that's the key to eliminating the serializer and
getting message delivery into the thread pool. I'd like to be able to
mark a queue so that it can wake an IO thread when it's got messages.
I don't think that anything you describe below needs non-IO wakeup of
the threads! Although time based wakeup might turn out to be useful I
have no plans to implement it the moment.
Not at all interested in time based wakeup. What I describe below is
wakeup based on presence of messages on a queue. How do we do that with
only IO based wakeup?
I think we can drive message delivery from IO wakeups like this:
Producer ready-to-read wakeup: only goes as far as enqueueing the message.
this is already how it works.
Currently the producer calls Queue::deliver which calls the Serializer.
I think we agree that we'd like to get rid of the serializer call at
this point.
Consumer thread ready-to-write wakeup: visits queues that it consumes
from and
- if it finds messages, write until writing would block, then go back
to sleep.
- if there's not enough data to block writing, mark myself as an
"unsatisfied consumer" on all queues of interest.
- Any queue with unsatisfied consumers enables IO wakeups.
Yes this would be a good way to make it work (I think this the general
scheme that Gordon and I have been pushing for a while now). The IO
framework already has a place to hook this into (via a yet-as-unwritten
callback).
Great. So do you or Gordon have plans to implement that in the short
term? If not I can look into it.
Queue-has-messages wakeup:
- write messages to unsatisfied consumers, writing until write would block.
- remove unsatisfied consumer marker for all consumers that are blocked
in write, leave it for any that are not.
What this needs to do is poke the IO framework to tell it that we are
now interested in writing on a specific connection.
Potentially a set of connections. There may be more than one consumer
interested in the queue.
so that if we
weren't looking for writeability before we will now. This is already
possible.
So are you saying that if a consumer is writable but there are no
messages to write it would mark itself "unsatisfied" on the queues it is
interested in and disable it's write trigger (or rather, not re-enable
its write trigger). When a queue has messages again it would enable the
write triggers of unsatified consumers causing their write threads to
wake up and contend for messages to write. That avoids the need for
non-IO wakeup. Is that what you mean? (presumably epoll is level
triggered for this to work)
The idea is that every consumer is always in exactly one of these states:
- blocked on write to network.
- blocked waiting for message from a set of queues.
- being serviced by a thread (which could be triggered by
ready-to-write or a queue-has-messages, but not both)
Does that make sense?
Cheers,
Alan.
