Anastasios Zouzias created SPARK-26121:
------------------------------------------
Summary: [Structured Streaming] Allow users to define prefix of
Kafka's consumer group (group.id)
Key: SPARK-26121
URL: https://issues.apache.org/jira/browse/SPARK-26121
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.4.0
Reporter: Anastasios Zouzias
I run in the following situation with Spark Structure Streaming (SS) using
Kafka.
In a project that I work on, there is already a secured Kafka setup where ops
can issue an SSL certificate per "[group.id|http://group.id/]";, which should be
predefined (or its prefix to be predefined).
On the other hand, Spark SS fixes the [group.id|http://group.id/] to
val uniqueGroupId =
s"spark-kafka-source-${UUID.randomUUID}-${metadataPath.hashCode}"
see, i.e.,
[https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L124]
https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L81
I guess Spark developers had a good reason to fix it, but is it possible to
make configurable the prefix of the above uniqueGroupId ("spark-kafka-source")?
The rational is that spark users are not forced to use the same certificate on
group-ids of the form (spark-kafka-source-*).
DoD:
* Allow spark SS users to define the group.id prefix as input parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
