HI, I have issue connecting to to kafka with SSL, tried a lot of options and stuck, maybe someone can suggest what is wrong here, on clickhouse side everything is fine:
[root@server.tld01 config]# cat connect-distributed.properties | grep -v ^# | grep -v '^$' group.id=connect-cluster key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=true offset.storage.topic=connect-offsets offset.storage.replication.factor=1 config.storage.topic=connect-configs config.storage.replication.factor=1 status.storage.topic=connect-status status.storage.replication.factor=1 offset.flush.interval.ms=10000 listeners=HTTP://127.0.0.1:8083 plugin.path=/opt/kafka/connectors ssl.keystore.location=/opt/kafka.client.p8.pem ssl.keystore.type=PEM bootstrap.servers=server.tld01:9091 security.protocol=SSL ssl.truststore.type=PEM ssl.truststore.location=/opt/kafka/kafka.ca.pem ssl.client.auth=required [root@server.tld01 config]# cat server.properties | grep -v ^# | grep -v '^$' process.roles=broker,controller node.id=281724871 controller.quorum.voters=131171308@server.tld01:9093,281724871@server.tld02 :9093,8884189@server.tld03:9093 listeners=BROKER://:9091,BROKERSASL://:9092,CONTROLLER://:9093 inter.broker.listener.name=BROKER sasl.enabled.mechanisms=SCRAM-SHA-512 controller.listener.names=CONTROLLER listener.security.protocol.map=BROKER:SSL,BROKERSASL:SASL_SSL,CONTROLLER:SSL authorizer.class.name =org.apache.kafka.metadata.authorizer.StandardAuthorizer super.users=User:CN=server;User:CN=client listener.name.broker.ssl.keystore.type=PEM listener.name.broker.ssl.keystore.location=/opt/kafka/kafka.server.p8.pem listener.name.broker.ssl.truststore.type=PEM listener.name.broker.ssl.truststore.location=/opt/kafka/kafka.ca.pem listener.name.broker.ssl.client.auth=required listener.name.brokersasl.ssl.keystore.type=PEM listener.name.brokersasl.ssl.keystore.location=/opt/kafka/kafka.server.p8.pem listener.name.brokersasl.ssl.client.auth=none listener.name.brokersasl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required; listener.name.controller.ssl.keystore.type=PEM listener.name.controller.ssl.keystore.location=/opt/kafka/kafka.server.p8.pem listener.name.controller.ssl.truststore.type=PEM listener.name.controller.ssl.truststore.location=/opt/kafka/kafka.ca.pem listener.name.controller.ssl.client.auth=required num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/var/lib/kafka-logs num.partitions=3 default.replication.factor=3 auto.create.topics.enable=true min.insync.replicas=2 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=3 share.coordinator.state.topic.replication.factor=3 share.coordinator.state.topic.min.isr=2 transaction.state.log.replication.factor=3 transaction.state.log.min.isr=2 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 curl -X GET http://localhost:8083/connectors curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{ "name": "clickhouse-sink-connector", "config": { "connector.class": "com.clickhouse.kafka.connect.ClickHouseSinkConnector", "tasks.max": "1", "consumer.override.max.poll.records": "5000", "consumer.override.max.partition.fetch.bytes": "5242880", "errors.retry.timeout": "60", "exactlyOnce": "false", "hostname": "clickhouse-db01.tld", "security.protocol": "SSL", "ssl": true, "ssl.truststore.location": "/opt/kafka/clickhouse.ca.pem", "ssl.truststore.type": "PEM", "port": "8443", "topics": "ticketTransactions.aggregator_dev.ticketTransactions", "username":"clickhouse-kafka", "password":"PASSS", "database":"DB", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "errors.log.enable": "true", "errors.log.include.messages": "true" } }' server.log multiple lines: [2025-07-23 11:06:26,535] INFO [SocketServer listenerType=BROKER, nodeId=281724871] Failed authentication with /127.0.0.1 (channelId=127.0.0.1:9091-127.0.0.1:52730-1-16786) (SSL handshake failed) (org.apache.kafka.common.network.Selector) connect.log multiple lines: [2025-07-23 11:12:49,230] WARN [clickhouse-sink-connector|task-0] [Consumer clientId=connector-consumer-clickhouse-sink-connector-0, groupId=connect-clickhouse-sink-connector] Bootstrap broker 127.0.0.1:9091 (id: -1 rack: null isFenced: false) disconnected (org.apache.kafka.clients.NetworkClient:1255) [2025-07-23 11:12:50,184] INFO [clickhouse-sink-connector|task-0] [Consumer clientId=connector-consumer-clickhouse-sink-connector-0, groupId=connect-clickhouse-sink-connector] Rebootstrapping with [/ 127.0.0.1:9091] (org.apache.kafka.clients.Metadata:314) [2025-07-23 11:12:51,032] INFO [clickhouse-sink-connector|task-0] [Consumer clientId=connector-consumer-clickhouse-sink-connector-0, groupId=connect-clickhouse-sink-connector] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:1072) [2025-07-23 11:12:51,032] INFO [clickhouse-sink-connector|task-0] [Consumer clientId=connector-consumer-clickhouse-sink-connector-0, groupId=connect-clickhouse-sink-connector] Cancelled in-flight API_VERSIONS request with correlation id 4490 due to node -1 being disconnected (elapsed time since creation: 30ms, elapsed time since send: 30ms, throttle time: 0ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient:411) kafka DEBUG says that kafka-connect is trying to connect without SSL: Jul 22 14:49:27 dkurmis kafka[3579826]: ) Jul 22 14:49:27 dkurmis kafka[3579826]: javax.net.ssl|DEBUG|73|data-plane-kafka-network-thread-281724871-ListenerName(BROKER)-SSL-1|2025-07-22 14:49:27.530 UTC|CertificateMessage.java:1172|Consuming client Certificate handshake message ( Jul 22 14:49:27 dkurmis kafka[3579826]: "Certificate": { Jul 22 14:49:27 dkurmis kafka[3579826]: "certificate_request_context": "", Jul 22 14:49:27 dkurmis kafka[3579826]: "certificate_list": [ Jul 22 14:49:27 dkurmis kafka[3579826]: ] Jul 22 14:49:27 dkurmis kafka[3579826]: } Jul 22 14:49:27 dkurmis kafka[3579826]: ) Jul 22 14:49:27 dkurmis kafka[3579826]: javax.net.ssl|ERROR|73|data-plane-kafka-network-thread-281724871-ListenerName(BROKER)-SSL-1|2025-07-22 14:49:27.530 UTC|TransportContext.java:375|Fatal (BAD_CERTIFICATE): Empty client certificate chain ( Jul 22 14:49:27 dkurmis kafka[3579826]: "throwable" : { Jul 22 14:49:27 dkurmis kafka[3579826]: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
