Hi All,
I'm trying to run MM2 in a Kafka connect cluster but I'm having some issues. I
had successfully run MM2 in a dedicated MM cluster but now haven a requirement
that means I have to run in a connect cluster in order to be able to support
additional connectors in the same cluster. I'm just trying to replicate a topic
from the source cluster to the target cluster at the moment.
Firstly I'm looking to understand which configuration goes into the
connect.distributed.properties file and which would go into the connector
configuration. The documentation here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RunningMirrorMakerinaConnectcluster
has the following example:
{ "name": "us-west-source", "connector.class":
"org.apache.kafka.connect.mirror.MirrorSourceConnector",
"source.cluster.alias": "us-west", "target.cluster.alias": "us-east",
"source.cluster.bootstrap.servers": "us-west-host1:9091", "topics": ".*"}
Is this all that's required? How will the MirrorSourceConnector know where to
send the records i.e. do we not have to specify a
target.cluster.bootstrap.servers list? It seems like if you don't specify a
target.cluster.bootstrap.servers property the connector seems to fail on
initialization of the target admin client since this property is required.
Currenlty I have the following in a connect.distributed.properites file which I
use to start one Kafka connect worker:
bootstrap.servers=localhost9092group.id=mygroupid
config.storage.replication.factor=1config.storage.topic=myconfigtopicoffset.storage.replication.factor=1offset.storage.topic=myoffsetstopicstatus.storage.replication.factor=1status.storage.topic=mystatustopic
replication.factor=1rest.port=8082
key.converter=org.apache.kafka.connect.storage.StringConvertervalue.converter=org.apache.kafka.connect.storage.StringConverterinternal.key.converter=org.apache.kafka.connect.json.JsonConverterinternal.value.converter=org.apache.kafka.connect.json.JsonConverter
Then I create a connector with the following config:
curl localhost:8082/connectors -X POST { "name": "replicate_testing1"
"config": { "connector.class":
"org.apache.kafka.connect.mirror.MirrorSourceConnector",
"clusters": "source, target",
"source.cluster.alias": "A",
"target.cluster.alias": "B",
"source.cluster.bootstrap.servers": "localhost:9092",
"target.cluster.bootstrap.servers":
"myserver1:9092,myserver2:9092,myserver3:9092",
"A->B.enabled": "true", "A->B.topics": "testing1",
"config.storage.replication.factor": "1",
"config.storage.topic": "myconfigtopic",
"offset.storage.replication.factor": "1",
"offset.storage.topic": "myoffsetstopic",
"status.storage.replication.factor": "1",
"status.storage.topic": "mystatustopic",
"checkpoints.topic.replication.factor": "1",
"heartbeats.topic.replication.factor" "1",
"offset-syncs.topic.replication.factor: "1",
"target.security.protocol": "SASL_SSL",
"target.sasl.kerberos.service.name": "kafka",
"target.sasl.jaas.config":
"com.sun.security.auth.module.lrb5LoginModule required useTicketCache=true"
}}
At the moment I'm just trying to replicate the "testing1" topic from the source
to the target cluster. The connector starts successfully but I can't replicate
from source to target.
Firstly, if someone could confirm this is the correct way in terms of
configuration to run MM in a connect cluster that would be great then I can
describe the issues I'm seeing.
Many Thanks,
Jamie
