[
https://issues.apache.org/jira/browse/KAFKA-12534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345886#comment-17345886
]
kaushik srinivas edited comment on KAFKA-12534 at 5/17/21, 5:52 AM:
--------------------------------------------------------------------
Hi,
We have tried the exact steps. Captured the commands and logs in detail. The
scenario to change the keystore password does not work still.
sequence of steps to reproduce
# install kafka broker by generating a CA, truststore and keystore. (password
for stores: 123456)
# re generate the keystore with a new password (1234567). Use the same old
generated CA and trust store from step1.
# issue the dynamic reconfig command after replacing the keystore file in the
specified location.
# dynamic config command issued:
{code:java}
./kafka-configs --bootstrap-server kafkabroker:9092 --command-config
ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config
'listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
{code}
Note: listener name is ssl and is in the format specified in
[https://docs.confluent.io/platform/current/kafka/dynamic-config.html#updating-ssl-keystore-of-an-existing-listener]
# command fails with following trace
{code:java}
Error while executing config command with args '--bootstrap-server
kafkabroker:9092 --command-config ssl.properties --entity-type brokers
--entity-name 1001 --alter --add-config
listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidRequestException: Invalid config value
for resource ConfigResource(type=BROKER, name='1001'): Invalid value
org.apache.kafka.common.config.ConfigException: Validation of dynamic config
update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to
load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for
configuration Invalid dynamic configuration
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:338)
at
kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:308)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:85)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid
config value for resource ConfigResource(type=BROKER, name='1001'): Invalid
value org.apache.kafka.common.config.ConfigException: Validation of dynamic
config update of SSLFactory failed: org.apache.kafka.common.KafkaException:
Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for
configuration Invalid dynamic configuration
{code}
Kafka broker logs the below output
{code:java}
{ "timezone":"UTC", "log":{"message":"data-plane-kafka-request-handler-5 -
kafka.server.AdminManager - [Admin Manager on Broker 1001]: Invalid config
value for resource ConfigResource(type=BROKER, name='1001'): Invalid value
org.apache.kafka.common.config.ConfigException: Validation of dynamic config
update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to
load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for
configuration Invalid dynamic configuration"}} {code}
As per docs, the CA is not supposed to be changed and we have maintained that
and the CA and trust stores are not changed. Also another observation is that,
when for example the country name in the cert generation is changed and the
certificate is regenerated, the dynamic config command works fine and we could
see the ssl certs being reloaded in the kafka broker logs.
But when the keystore password is changed, things have never worked for us even
after so many attempts of retries. Can you please help in reproducing this
issue and provide some detailed steps if possible for the case where the
keystore's password is being changed ? It has clearly never worked for us, even
after many attempts.
was (Author: kaushik srinivas):
Hi,
We have tried the exact steps. Captured the commands and logs in detail. The
scenario to change the keystore password does not work still.
sequence of steps to reproduce
# install kafka broker by generating a CA, truststore and keystore. (password
for stores: 123456)
# re generate the keystore with a new password (1234567). Use the same old
generated CA and trust store from step1.
# issue the dynamic reconfig command after replacing the keystore file in the
specified location.
# dynamic config command issued:
{code:java}
./kafka-configs --bootstrap-server kafkabroker:9092 --command-config
ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config
'listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
{code}
Note: listener name is ssl and is in the format specified in
[https://docs.confluent.io/platform/current/kafka/dynamic-config.html#updating-ssl-keystore-of-an-existing-listener]
# command fails with following trace
{code:java}
Error while executing config command with args '--bootstrap-server
kafkabroker:9092 --command-config ssl.properties --entity-type brokers
--entity-name 1001 --alter --add-config
listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidRequestException: Invalid config value
for resource ConfigResource(type=BROKER, name='1001'): Invalid value
org.apache.kafka.common.config.ConfigException: Validation of dynamic config
update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to
load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for
configuration Invalid dynamic configuration
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:338)
at
kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:308)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:85)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid
config value for resource ConfigResource(type=BROKER, name='1001'): Invalid
value org.apache.kafka.common.config.ConfigException: Validation of dynamic
config update of SSLFactory failed: org.apache.kafka.common.KafkaException:
Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for
configuration Invalid dynamic configuration
{code}
Kafka broker logs the below output
{code:java}
{ "timezone":"UTC", "log":{"message":"data-plane-kafka-request-handler-5 -
kafka.server.AdminManager - [Admin Manager on Broker 1001]: Invalid config
value for resource ConfigResource(type=BROKER, name='1001'): Invalid value
org.apache.kafka.common.config.ConfigException: Validation of dynamic config
update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to
load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for
configuration Invalid dynamic configuration"}}
{code}
{code:java}
{code}
> kafka-configs does not work with ssl enabled kafka broker.
> ----------------------------------------------------------
>
> Key: KAFKA-12534
> URL: https://issues.apache.org/jira/browse/KAFKA-12534
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 2.6.1
> Reporter: kaushik srinivas
> Priority: Critical
>
> We are trying to change the trust store password on the fly using the
> kafka-configs script for a ssl enabled kafka broker.
> Below is the command used:
> kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers
> --entity-name 1001 --alter --add-config 'ssl.truststore.password=xxx'
> But we see below error in the broker logs when the command is run.
> {"type":"log", "host":"kf-2-0", "level":"INFO",
> "neid":"kafka-cfd5ccf2af7f47868e83473408", "system":"kafka",
> "time":"2021-03-23T12:14:40.055", "timezone":"UTC",
> "log":\{"message":"data-plane-kafka-network-thread-1002-ListenerName(SSL)-SSL-2
> - org.apache.kafka.common.network.Selector - [SocketServer brokerId=1002]
> Failed authentication with /127.0.0.1 (SSL handshake failed)"}}
> How can anyone configure ssl certs for the kafka-configs script and succeed
> with the ssl handshake in this case ?
> Note :
> We are trying with a single listener i.e SSL:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
