[
https://issues.apache.org/jira/browse/SPARK-26322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715621#comment-16715621
]
ASF GitHub Bot commented on SPARK-26322:
----------------------------------------
vanzin commented on a change in pull request #23274: [SPARK-26322][SS] Add
spark.kafka.token.sasl.mechanism to ease delegation token configuration.
URL: https://github.com/apache/spark/pull/23274#discussion_r240393933
##########
File path: docs/structured-streaming-kafka-integration.md
##########
@@ -669,144 +669,7 @@ Kafka broker configuration):
After obtaining delegation token successfully, Spark distributes it across
nodes and renews it accordingly.
Delegation token uses `SCRAM` login module for authentication and because of
that the appropriate
-`sasl.mechanism` has to be configured on source/sink (it must match with Kafka
broker configuration):
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-
-// Setting on Kafka Source for Streaming Queries
-val df = spark
- .readStream
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("subscribe", "topic1")
- .load()
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .as[(String, String)]
-
-// Setting on Kafka Source for Batch Queries
-val df = spark
- .read
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("subscribe", "topic1")
- .load()
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .as[(String, String)]
-
-// Setting on Kafka Sink for Streaming Queries
-val ds = df
- .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .writeStream
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("topic", "topic1")
- .start()
-
-// Setting on Kafka Sink for Batch Queries
-val ds = df
- .selectExpr("topic1", "CAST(key AS STRING)", "CAST(value AS STRING)")
- .write
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .save()
-
-{% endhighlight %}
-</div>
-<div data-lang="java" markdown="1">
-{% highlight java %}
-
-// Setting on Kafka Source for Streaming Queries
-Dataset<Row> df = spark
- .readStream()
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("subscribe", "topic1")
- .load();
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)");
-
-// Setting on Kafka Source for Batch Queries
-Dataset<Row> df = spark
- .read()
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("subscribe", "topic1")
- .load();
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)");
-
-// Setting on Kafka Sink for Streaming Queries
-StreamingQuery ds = df
- .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .writeStream()
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("topic", "topic1")
- .start();
-
-// Setting on Kafka Sink for Batch Queries
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
- .write()
- .format("kafka")
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512")
- .option("topic", "topic1")
- .save();
-
-{% endhighlight %}
-</div>
-<div data-lang="python" markdown="1">
-{% highlight python %}
-
-// Setting on Kafka Source for Streaming Queries
-df = spark \
- .readStream \
- .format("kafka") \
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
- .option("subscribe", "topic1") \
- .load()
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
-
-// Setting on Kafka Source for Batch Queries
-df = spark \
- .read \
- .format("kafka") \
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
- .option("subscribe", "topic1") \
- .load()
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
-
-// Setting on Kafka Sink for Streaming Queries
-ds = df \
- .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") \
- .writeStream \
- .format("kafka") \
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
- .option("topic", "topic1") \
- .start()
-
-// Setting on Kafka Sink for Batch Queries
-df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") \
- .write \
- .format("kafka") \
- .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
- .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
- .option("topic", "topic1") \
- .save()
-
-{% endhighlight %}
-</div>
-</div>
+`spark.kafka.token.sasl.mechanism` (default: `SCRAM-SHA-512`) has to be
configured (it must match with Kafka broker configuration).
Review comment:
The sentence is too long now; move the stuff in parentheses to a new
sentence. Also, "...must match the Kafka..."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Simplify kafka delegation token sasl.mechanism configuration
> ------------------------------------------------------------
>
> Key: SPARK-26322
> URL: https://issues.apache.org/jira/browse/SPARK-26322
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.0.0
> Reporter: Gabor Somogyi
> Priority: Major
>
> When Kafka delegation token obtained, SCRAM sasl.mechanism has to be
> configured for authentication. This can be configured on the related
> source/sink which is inconvenient from user perspective. Such granularity is
> not required and this configuration can be implemented with one central
> parameter.
> Kafka now supports 2 SCRAM related sasl.mechanism:
> - SCRAM-SHA-256
> - SCRAM-SHA-512
> and these are configured on brokers, which makes this configuration global.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
