gaborgsomogyi commented on a change in pull request #24305: [SPARK-27294][SS]
Add multi-cluster Kafka delegation token
URL: https://github.com/apache/spark/pull/24305#discussion_r279312918
##########
File path: docs/structured-streaming-kafka-integration.md
##########
@@ -703,11 +703,98 @@ Kafka broker configuration):
After obtaining delegation token successfully, Spark distributes it across
nodes and renews it accordingly.
Delegation token uses `SCRAM` login module for authentication and because of
that the appropriate
-`spark.kafka.sasl.token.mechanism` (default: `SCRAM-SHA-512`) has to be
configured. Also, this parameter
+`spark.kafka.clusters.${cluster}.sasl.token.mechanism` (default:
`SCRAM-SHA-512`) has to be configured. Also, this parameter
must match with Kafka broker configuration.
When delegation token is available on an executor it can be overridden with
JAAS login configuration.
+#### Configuration
+
+Delegation tokens can be obtained from multiple clusters and
<code>${cluster}</code> is an arbitrary unique identifier which helps to group
different configurations.
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ <tr>
+ <td><code>spark.kafka.clusters.${cluster}.bootstrap.servers</code></td>
+ <td>None</td>
+ <td>
+ A list of coma separated host/port pairs to use for establishing the
initial connection
+ to the Kafka cluster. For further details please see kafka
documentation. Only used to obtain delegation token.
+ </td>
+ </tr>
+ <tr>
+
<td><code>spark.kafka.clusters.${cluster}.target.bootstrap.servers.regex</code></td>
+ <td>.*</td>
+ <td>
+ When delegation token obtained it can be used on sources/sinks. If any
of the sources/sinks <code>bootstrap.servers</code> configuration
Review comment:
Yeah, its better wording. Added secure + unsecure together explanation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
