Jay Kreps created KAFKA-659:
-------------------------------
Summary: Support request pipelining in the network server
Key: KAFKA-659
URL: https://issues.apache.org/jira/browse/KAFKA-659
Project: Kafka
Issue Type: New Feature
Reporter: Jay Kreps
Currently the network layer in kafka will only process a single request at a
time from a given connection. The protocol is designed to allow pipelining of
requests which would improve latency.
There are two changes that would have to made for this to work, in my
understanding:
1. Currently once a completed request is read from a socket the server does not
register for "read interest" again until a response is sent. The server would
have to register for read interest immediately to allow reading more requests.
2. Currently the socket server adds all requests to a single "request channel"
that serves as a work queue for all the background i/o threads. One requirement
for Kafka is to do in order processing of requests from a given socket. This is
currently achieved by not reading any new requests from a socket until the
currently outstanding request is processed. To maintain this guarantee we would
have to guarantee that all requests from a particular socket went to the same
I/O thread. A simple way to do this would be to have work queue per I/O thread.
One downside of this is that pinning requests to I/O threads will add latency
variance--if that thread stalls due to a slow I/O no other thread can pick up
the slack. So perhaps there is a better way that isn't overly complex?
Would be good to nail down the design for this as a first step.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
