Thank you Neha, Allocating dedicated connection for fetching client worked fine and did not require any changes to kafka client library code.
Thinking about it, server perhaps could process Fetch and Produce requests in parallel, while still returning result in order as in the spec. What needs to be prevented is multiple producers or multiple fetchers simultaneous processing. I hope I'll have time to play with server a little bit. Vadim. On Thu, Sep 25, 2014 at 12:13 PM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > "The server guarantees that on a single TCP connection, requests will be > processed in the order they are sent and responses will return in that > order as well" > ==== > But to achieve proper functioning of the client, I need to allocate another > tcp connection for listeners now? > > Yes. The easiest way to send and receive is to use a producer client and a > consumer client, each of which will have it's own TCP connection to the > server. > > On Thu, Sep 25, 2014 at 11:40 AM, Vadim Chekan <kot.bege...@gmail.com> > wrote: > > > Hi all, > > > > I'm working on my own kafka client implementation and I noticed strange > > situation. > > > > Time(s) | Action > > 0.0 MetadataRequest > > 0.0 MetadataResponse > > 0.2 OffsetRequest > > 0.2 OffsetResponse > > 0.3 FetchRequest(MaxWaitTime=20sec) > > 6.0 ProduceRequest > > 31.0 FetchResponse > > ^^^^ notice 25sec gap! > > 31.0 ProduceResponse > > > > What is important, it seems to me that Fetch Request with long poll will > > block processing of all following requests for the duration of timeout, > > given that there is no new data. But new data *are* flowing in and > Produce > > Request is waiting right behind the Fetch one in server side processing > > queue. > > > > If what I describe is correct then I see the following problems. > > 1. From client's point of view, if they have a listener and a sender, > then > > sending data should produce immediate event on listener. > > > > 2. If I want clean shutdown of my application within 5sec, but my > listeners > > are configured with 20sec timeout, than I am risking losing the data. My > > application shutdown time now becomes at least as long as listener's > > pooling time, which is unreasonable. > > > > I do understand why it was implemented that way, probably because of the > > specification saying > > ==== > > "The server guarantees that on a single TCP connection, requests will be > > processed in the order they are sent and responses will return in that > > order as well" > > ==== > > But to achieve proper functioning of the client, I need to allocate > another > > tcp connection for listeners now? > > > > Also, I'm aware about new protocol proposal for the next version of > Kafka, > > would this issue be resolved there? > > > > Thanks, > > Vadim. > > > > -- > > From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is > > explicitly specified > > > -- >From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is explicitly specified
