[
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750328#comment-13750328
]
Chris Riccomini commented on KAFKA-1012:
----------------------------------------
Regarding transactionality, as described in
https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management, how do you
guarantee that the offset messages in a single transaction will be sequential
with no gaps? As far as I understand, your assumption is that if a transaction
has 5 messages, they will be materialized into the topic/partition as M1, M2,
M3, M4, M5, with no messages in between.
Is there a single-writer that's making this guarantee? I don't see how this is
possible if you allow clients to send produce requests directly to the topic.
Without a single writer, you could end up with M1, M2, .. other offset messages
.., M3, M4, M5. In such a case, how do you determine if the transaction has
failed? Is there some timeout (If I don't get all 5 messages within 60s, the
transaction is rolled back)?
> Implement an Offset Manager and hook offset requests to it
> ----------------------------------------------------------
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Priority: Minor
> Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets
> from brokers. Currently, consumers are not using this API and directly
> talking with Zookeeper.
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing,
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
