Henrique Mota created KAFKA-15841:
-------------------------------------
Summary: Add Support for Topic-Level Partitioning in Kafka Connect
Key: KAFKA-15841
URL: https://issues.apache.org/jira/browse/KAFKA-15841
Project: Kafka
Issue Type: Improvement
Components: connect
Reporter: Henrique Mota
In our organization, we utilize JDBC sink connectors to consume data from
various topics, where each topic is dedicated to a specific tenant with a
single partition. Recently, we developed a custom sink based on the standard
JDBC sink, enabling us to pause consumption of a topic when encountering
problematic records.
However, we face limitations within Kafka Connect, as it doesn't allow for
appropriate partitioning of topics among workers. We attempted a workaround by
breaking down the topics list within the 'topics' parameter. Unfortunately,
Kafka Connect overrides this parameter after invoking the {{taskConfigs(int
maxTasks)}} method from the {{org.apache.kafka.connect.connector.Connector}}
class.
We request the addition of support in Kafka Connect to enable the partitioning
of topics among workers without requiring a fork. This enhancement would
facilitate better load distribution and allow for more flexible configurations,
particularly in scenarios where topics are dedicated to different tenants.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
