I am working on migrating from Camus to Kafka Connect. I am working on the
implementation of Kafka Connect and specifically focused on distributed mode. I
am able to start a worker successfully on my local machine which I assume
communicates with my Kafka cluster. I am further able to run two GET commands
such as / and /connector-plugins which return the correct JSON. However, when I
try to POST a command to create a connector, I receive a 500 error and a time
out. Specifically, I use this command to POST for testing:
curl -X POST -H "Content-Type: application/json" --data '{"name":
"local-file-sink", "config": {"connector.class":"FileStreamSinkConnector",
"tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic" }}'
localhost:8083/connectors
and eventually I get this response:
{"error_code": 500, "message": "Request timed out"}
I am lost as to what is going on. The logs from my Kafka Connect distributed
worker show this:
[2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32 +0000]
"GET /connector-plugins HTTP/1.1" 200 315 2
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25 +0000]
"GET /connector-plugins HTTP/1.1" 200 315 3
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28 +0000]
"GET /connector-plugins HTTP/1.1" 200 315 2
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:07:38,644] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:06:08 +0000]
"GET /connectors HTTP/1.1" 500 48 90003
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:07:44,450] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:07:44 +0000]
"GET /connector-plugins HTTP/1.1" 200 315 1
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:13:06,703] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:11:36 +0000]
"POST /connectors HTTP/1.1" 500 48 90003
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:15:38,506] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:14:08 +0000]
"POST /connectors HTTP/1.1" 500 48 90005
(org.apache.kafka.connect.runtime.rest.RestServer:60)
where you can see the error codes and the commands.
I guess my main questions and issues are:
1. How can I better debug Kafka Connect so I can try and fix this?
2. Is there anything that I'm doing that is glaringly wrong?
3. Is there any step-by-step documentation or blog posts on getting a Kafka
Connect distributed worker and connector to run? I have not really seen
anything or even best practices kinds of documentation? Maybe I am just way too
early of an adopter.
I look forward to hearing back from the community and thank you for your help!
