Greg Harris created KAFKA-15575:
-----------------------------------
Summary: Prevent Connectors from exceeding tasks.max configuration
Key: KAFKA-15575
URL: https://issues.apache.org/jira/browse/KAFKA-15575
Project: Kafka
Issue Type: Task
Components: KafkaConnect
Reporter: Greg Harris
The Connector::taskConfigs(int maxTasks) function is used by Connectors to
enumerate tasks configurations. This takes an argument which comes from the
tasks.max connector config. This is the Javadoc for that method:
{noformat}
/**
* Returns a set of configurations for Tasks based on the current configuration,
* producing at most {@code maxTasks} configurations.
*
* @param maxTasks maximum number of configurations to generate
* @return configurations for Tasks
*/
public abstract List<Map<String, String>> taskConfigs(int maxTasks);
{noformat}
This includes the constraint that the number of tasks is at most maxTasks, but
this constraint is not enforced by the framework.
We should begin enforcing this constraint by dropping configs that exceed the
limit, and logging a warning. For sink connectors this should harmlessly
rebalance the consumer subscriptions onto the remaining tasks. For source
connectors that distribute their work via task configs, this may result in an
interruption in data transfer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
