[
https://issues.apache.org/jira/browse/KAFKA-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106190#comment-17106190
]
Joost van de Wijgerd commented on KAFKA-9953:
---------------------------------------------
Hi [~guozhang] thanks for the information, Indeed I see that the SerDes pass in
the topic so this can indeed be used to select the proper deserializer in that
case. I guess the typing on the Consumer and ConsumerRecords interface would
have to be Object though as I am now mixing diffent java class types.
The other situation (b) is a bit more problematic for us. We have a constant
stream of DomainEvents which are processed exactly once. If we would like to
process an event from that stream (one that was already there but we weren't
interested before) then rewinding the existing consumer would be hard to do
(not impossible of course) and it would block processing of newer events until
we have caught up again. In this scenario, adding a new consumer(group) would
be much simpler to implement (at the expense of network traffic of course, but
for us this is not an issue).
Arguably this could all be solved with a single consumer -> producer mapping
but unfortunately this is not the route we took. Since we are processing
financial transactions with this system I am very wary of rewriting our
consumer logic as we would have to drop our current consumer groups and create
a new one (for the whole service) and if we are not careful we might miss
transactions . Another solution would be to pair a producer to each consumer
but that will add extra overhead as well.
Maybe our use case / implementation is a bit out of the ordinary but fact is
that the kafka-clients library allows this construct and it also works. Except
for the fact that a latency of around 105ms is introduced with the standards
settings because the TransactionManager has to keep refreshing the
groupCoordinator Node. So best practices notwithstanding I think it makes sense
for the kafka-clients library to support many-to-one consumer -> producer
mappings in a performant way. If you decide not to, then maybe add at least a
warn log when this situation is detected inside the TransactionManager to alert
developers they are doing something wrong that will hurt the performance of
their app.
Obviously my preference is to have this fix in the kafka-clients library ;)
> support multiple consumerGroupCoordinators in TransactionManager
> ----------------------------------------------------------------
>
> Key: KAFKA-9953
> URL: https://issues.apache.org/jira/browse/KAFKA-9953
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Affects Versions: 2.5.0
> Reporter: Joost van de Wijgerd
> Priority: Major
> Attachments: KAFKA-9953.patch
>
>
> We are using kafka with a transactional producer and have the following use
> case:
> 3 KafkaConsumers (each with their own ConsumerGroup) polled by the same
> thread and 1 transactional kafka producer. When we add the offsets to the
> transaction we run into the following problem:
> TransactionManager only keeps track of 1 consumerGroupCoordinator, however it
> can be that some consumerGroupCoordinators are on another node, now we
> constantly see the TransactionManager switching between nodes, this has
> overhead of 1 failing _TxnOffsetCommitRequest_ and 1 unnecessary
> _FindCoordinatorRequest_.
> Also with _retry.backoff.ms_ set to 100 by default this is causing a pause
> of 100ms for every other transaction (depending on what KafkaConsumer
> triggered the transaction of course)
> If the TransactionManager could keep track of coordinator nodes per
> consumerGroupId this problem would be solved.
> I have already a patch for this but still need to test it. Will add it to the
> ticket when that is done
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
