gaborgsomogyi commented on a change in pull request #23274: [SPARK-26322][SS]
Add spark.kafka.token.sasl.mechanism to ease delegation token configuration.
URL: https://github.com/apache/spark/pull/23274#discussion_r240528956
##########
File path: docs/structured-streaming-kafka-integration.md
##########
@@ -669,144 +669,7 @@ Kafka broker configuration):
After obtaining delegation token successfully, Spark distributes it across
nodes and renews it accordingly.
Delegation token uses `SCRAM` login module for authentication and because of
that the appropriate
-`sasl.mechanism` has to be configured on source/sink (it must match with Kafka
broker configuration):
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-
-// Setting on Kafka Source for Streaming Queries
-val df = spark
- .readStream
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("subscribe", "topic1")
- .load()
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .as[(String, String)]
-
-// Setting on Kafka Source for Batch Queries
-val df = spark
- .read
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("subscribe", "topic1")
- .load()
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .as[(String, String)]
-
-// Setting on Kafka Sink for Streaming Queries
-val ds = df
- .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .writeStream
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("topic", "topic1")
- .start()
-
-// Setting on Kafka Sink for Batch Queries
-val ds = df
- .selectExpr("topic1", "CAST(key AS STRING)", "CAST(value AS STRING)")
- .write
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .save()
-
-{% endhighlight %}
-</div>
-<div data-lang="java" markdown="1">
-{% highlight java %}
-
-// Setting on Kafka Source for Streaming Queries
-Dataset<Row> df = spark
- .readStream()
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("subscribe", "topic1")
- .load();
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)");
-
-// Setting on Kafka Source for Batch Queries
-Dataset<Row> df = spark
- .read()
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("subscribe", "topic1")
- .load();
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)");
-
-// Setting on Kafka Sink for Streaming Queries
-StreamingQuery ds = df
- .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .writeStream()
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("topic", "topic1")
- .start();
-
-// Setting on Kafka Sink for Batch Queries
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .write()
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("topic", "topic1")
- .save();
-
-{% endhighlight %}
-</div>
-<div data-lang="python" markdown="1">
-{% highlight python %}
-
-// Setting on Kafka Source for Streaming Queries
-df = spark \
- .readStream \
- .format("kafka") \
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
- .option("subscribe", "topic1") \
- .load()
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
-
-// Setting on Kafka Source for Batch Queries
-df = spark \
- .read \
- .format("kafka") \
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
- .option("subscribe", "topic1") \
- .load()
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
-
-// Setting on Kafka Sink for Streaming Queries
-ds = df \
- .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") \
- .writeStream \
- .format("kafka") \
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
- .option("topic", "topic1") \
- .start()
-
-// Setting on Kafka Sink for Batch Queries
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") \
- .write \
- .format("kafka") \
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
- .option("topic", "topic1") \
- .save()
-
-{% endhighlight %}
-</div>
-</div>
+`spark.kafka.token.sasl.mechanism` (default: `SCRAM-SHA-512`) has to be
configured (it must match with Kafka broker configuration).
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
