gaborgsomogyi commented on a change in pull request #24305: [SPARK-27294][SS]
Add multi-cluster Kafka delegation token
URL: https://github.com/apache/spark/pull/24305#discussion_r280744107
##########
File path: docs/structured-streaming-kafka-integration.md
##########
@@ -703,11 +703,99 @@ Kafka broker configuration):
After obtaining delegation token successfully, Spark distributes it across
nodes and renews it accordingly.
Delegation token uses `SCRAM` login module for authentication and because of
that the appropriate
-`spark.kafka.sasl.token.mechanism` (default: `SCRAM-SHA-512`) has to be
configured. Also, this parameter
+`spark.kafka.clusters.${cluster}.sasl.token.mechanism` (default:
`SCRAM-SHA-512`) has to be configured. Also, this parameter
must match with Kafka broker configuration.
When delegation token is available on an executor it can be overridden with
JAAS login configuration.
+#### Configuration
+
+Delegation tokens can be obtained from multiple clusters and
<code>${cluster}</code> is an arbitrary unique identifier which helps to group
different configurations.
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ <tr>
+ <td><code>spark.kafka.clusters.${cluster}.bootstrap.servers</code></td>
Review comment:
As I'm thinking about this maybe it would be better to call it
`auth.bootstrap.servers` since `bootstrap.servers` in Kafka has a defined
format and can be a subset of all brokers. This way the name would tell both
(only auth + format). What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
