[
https://issues.apache.org/jira/browse/KAFKA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151896#comment-13151896
]
Jay Kreps commented on KAFKA-208:
---------------------------------
Yes, but I was actually hoping to even do a multifetch version. So it would be
request A:a, B:b, C:c, timeout:500ms
immediately get data:B:b', C:c'
request A:a, B:b', C:c', timeout:500ms
no data for any topics, request blocks
message on C
get data for C:c''
request A:a, B:b', C:c'', timeout:500ms
Etc.
I think this multifetch approach simplifies things for the consumer
implementation. So I think that might cover what you want, I think the primary
differences is that there is no separate registration of interest and polling
for data, which i think makes sense.
The only question I think was left open was whether, in addition to a timeout,
we could also support a min_bytes parameter so that the response would wait for
at least that much data to accumulate. This is slightly more complex to
implement on the server side but could improve efficiency.
> Efficient polling mechanism for many topics
> -------------------------------------------
>
> Key: KAFKA-208
> URL: https://issues.apache.org/jira/browse/KAFKA-208
> Project: Kafka
> Issue Type: New Feature
> Reporter: Taylor Gautier
>
> Currently, the way to poll many topics is to submit a request for each one in
> turn, and read the responses. Especially if there are few messages delivered
> on many topics, the network overhead to implement this scheme can far
> outweigh the bandwidth of the actual messages delivered.
> Effectively, for topics A, B, C the request/response scheme is the following:
> -> Request A offset a
> -> Request B offset b
> -> Request C offset c
> <- no messages
> <- 1 message offset b
> <- no messages
> -> Request A offset a
> -> Request B offset b'
> -> Request C offset c
> <- no messages
> <- no messages
> <- no messages
> etc.
> I propose a more efficient mechanism which works a bit like epoll in that the
> client can register interest in a particular topic. There are many options
> for the implementation, but the one I suggest goes like so:
> -> Register interest A offset a
> -> Register interest B offset b
> -> Register interest C offset c
> -> Next message (null)
> <- messages for B (1 message)
> -> Next message (topic B offset b')
> <- no messages
> -> Unregister Interest C
> ...
> It is possible to implement the "Next Message" request as either blocking or
> non-blocking. I suggest that the request format include space for the
> timeout, which if set to 0 will be a nonblocking response, and if set to
> anything other than 0, will block for at most timeout ms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
