[
https://issues.apache.org/jira/browse/KAFKA-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938692#comment-15938692
]
Stephen Durfey commented on KAFKA-4159:
---------------------------------------
So, full disclosure, I believe my original desire for overriding the group.id
is alleviated once I can upgrade to kafka 0.10.2.0 and use single message
transforms.
The current use case for me was pulling data from a sql server database over
jdbc to push into kafka. In the platform the data is organized into topics with
a source identifier in the topic name with our own wrapper avro for the payload
to provide additional metadata about the data coming from the database. Because
the data needed to be wrapped I wasn't able to write directly to these topics
with kafka connect, so they needed to be written to different topics. I had
additional connectors pulling from those topics to transform the data into the
format I needed and had a custom sink connector write out the wrapped paylods.
To generate the metadata it was specific to a table, so the design had a
transformation connector per topic that kafka connect wrote to.
I wanted to be able to provide progress of transforming this data through the
[kafka offset monitor|https://github.com/quantifind/KafkaOffsetMonitor], but
having a connector per topic with a unique group.id per connector really made
the offset monitor almost useless, as there were just way too many consumer
groups, and as we scaled for more databases, it was only going to get worse.
Since all the tables belonged to one source, I wanted to be able to group all
the topics for that one source into a single identifier. That way I could just
click on that identifier in the offset monitor and view the progress for all
topics for that particular source.
I could've overridden the group.id at the worker level and that would work as
well, as long as the connect instance was only ever going to be for that
source. However, I think long term that wasn't going to be the case.
> Allow overriding producer & consumer properties at the connector level
> ----------------------------------------------------------------------
>
> Key: KAFKA-4159
> URL: https://issues.apache.org/jira/browse/KAFKA-4159
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect
> Reporter: Shikhar Bhushan
> Assignee: Stephen Durfey
>
> As an example use cases, overriding a sink connector's consumer's partition
> assignment strategy.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
