[
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744718#comment-13744718
]
Tejas Patil commented on KAFKA-1012:
------------------------------------
(1) Currently, "KafkaApis" has to figure out the leader of offsets topic and
perform forwarding of the OffsetFetchRequest. From an implementation
perspective, "OffsetManager" must abstract out those things. Also, another bad
thing is that in case of Zk based offset manager, offset fetch request may be
get redirected even if not required. Both these could be resolved by moving a
things to "OffsetManager" but as there is strong coupling (leaderCache,
offsetFetchRequest, aliveBrokers) the entire change might make things look
hideous.
(2) Is it necessary to have "offset topic" and save the offsets to logs while
using Zk based offset manager ? Technically if a user just wants to continue
using Zk based offset manager, having "offsets topic" would be an overhead. On
the other hand, having logs for the "offsets topic" may end up giving room for
smooth transition across the 2 offset manager implementations.
(3) There is no purgatory maintained for Offset fetch request as fetch requests
would end up querying Zk or reading from in-memory table of broker (with at max
one extra hop). This does not involve any disk access and is expected to be
fast. It could be argued that a busy broker might not be able to respond to the
request making the request to time out. The request handler threads of the
broker, which forwards the request, might not process any requests in the mean
time. If the current implementation seems spooky in this aspect, some
alternatives would be: add purgatory for offset fetch request (seems overkill
to me) OR use a thick client which would directly contact the leader directly.
Any suggestions for the items above would be highly appreciated.
> Implement an Offset Manager and hook offset requests to it
> ----------------------------------------------------------
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Priority: Minor
> Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets
> from brokers. Currently, consumers are not using this API and directly
> talking with Zookeeper.
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing,
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
