Yash Mayya created KAFKA-15470:
----------------------------------
Summary: Allow creating connectors in a stopped state
Key: KAFKA-15470
URL: https://issues.apache.org/jira/browse/KAFKA-15470
Project: Kafka
Issue Type: New Feature
Components: connect, KafkaConnect
Reporter: Yash Mayya
Assignee: Yash Mayya
Fix For: 3.7.0
[KIP-875: First-class offsets support in Kafka
Connect|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect]
introduced a new {{STOPPED}} state for connectors along with some REST API
endpoints to retrieve and modify offsets for connectors. Currently, only
connectors that already exist can be stopped and any newly created connector
will always be in the {{RUNNING}} state initially. Allowing the creation of
connectors in a {{STOPPED}} state will facilitate multiple new use cases. One
interesting use case would be to migrate connectors from one Kafka Connect
cluster to another. Individual connector migration would be useful in a number
of scenarios such as breaking a large cluster into multiple smaller clusters
(or vice versa), moving a connector from a cluster running in one data center
to another etc. A connector migration could be achieved by using the following
sequence of steps :-
# Stop the running connector on the original Kafka Connect cluster
# Retrieve the offsets for the connector via the {{GET
/connectors/\{connector}/offsets}} endpoint
# Create the connector in a stopped state using the same configuration on the
new Kafka Connect cluster
# Alter the offsets for the connector on the new cluster via the {{PATCH
/connectors/\{connector}/offsets}} endpoint (using the offsets obtained from
the original cluster)
# Resume the connector on the new cluster and delete it on the original cluster
Another use case for creating connectors in a stopped state could be deploying
connectors as a part of a larger data pipeline before the source / sink data
system has been created or is ready for data transfer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
