xiangfu0 commented on issue #19184:
URL: https://github.com/apache/pinot/issues/19184#issuecomment-5335842328
Shipped in #19202, merged as `fda09b7a70` on `master`. Docs landed in
pinot-contrib/pinot-docs#994.
**What works now**
Declare the provider namespace and reference secrets by file in
`streamConfigs`; the values stay out of ZooKeeper and are resolved by Kafka
when the client is constructed:
```json
{
"streamConfigs": {
"streamType": "kafka",
"security.protocol": "SSL",
"config.providers": "file",
"config.providers.file.class":
"org.apache.kafka.common.config.provider.FileConfigProvider",
"config.providers.file.param.allowed.paths": "/vault/secrets",
"ssl.keystore.location": "/vault/secrets/kafka-keystore.p12",
"ssl.keystore.password":
"$${file:/vault/secrets/kafka-keypass.txt:keystore.password}",
"ssl.truststore.location": "/vault/secrets/kafka-truststore.p12",
"ssl.truststore.password":
"$${file:/vault/secrets/kafka-trustpass.txt:truststore.password}"
}
}
```
Both the server consumer path and the controller AdminClient path carry the
provider namespace through property filtering, for the Kafka 3.x and 4.x
connectors.
**Two corrections to the original report**
1. **The `stream.kafka.consumer.prop.*` prefix does not work.**
`KafkaStreamConfigProperties.KAFKA_CONSUMER_PROP_PREFIX` is declared but
referenced nowhere; `buildProperties()` copies `streamConfigsMap` verbatim with
no prefix stripping, so prefixed keys are dropped by the client filter. Use
plain Kafka names such as `ssl.keystore.password`. The config example in the
description above will not work as written.
2. **Provider references need the `$$` escape.** Pinot's own
`${name:default}` substitution runs over table configs before Kafka ever sees
them, so a bare `${file:/path:key}` was being consumed by Pinot and silently
replaced with its "default" — the path fragment after the first colon.
`$${...}` passes the reference through to Kafka intact. This is why the feature
could not simply work by allow-listing `config.providers`.
**Additional notes**
- `config.providers.<alias>.param.allowed.paths` is **required** for
`FileConfigProvider` and enforced in code. An unrestricted provider would let
anyone who can write a table config read arbitrary files readable by the server
or controller process.
- Rotation takes effect when a Kafka client is recreated (`forceCommit` /
segment completion). The shared AdminClient keeps its credentials until all
references are released.
- Provider references cannot be combined with
`stream.kafka.ssl.server.certificate` / `stream.kafka.ssl.client.certificate`
auto-generated stores; this is rejected up front.
**Not shipped:** item 4 of the proposed solution — hot-reloading
keystore/truststore files on *already running* consumers without `forceCommit`.
That remains out of scope. Please open a separate issue if you need it.
Thanks @sudheesh-87 for the detailed report and @goutamadwant for the
implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
