This makes sense to me and was going to propose something similar in order to
be able to use the kafka acls more effectively as well, can you file a jira for
it?
Tom
On Friday, November 9, 2018, 2:26:12 AM CST, Anastasios Zouzias
<zouz...@gmail.com> wrote:
Hi all,
I run in the following situation with Spark Structure Streaming (SS) using
Kafka.
In a project that I work on, there is already a secured Kafka setup where ops
can issue an SSL certificate per "group.id", which should be predefined (or
hopefully its prefix to be predefined).
On the other hand, Spark SS fixes the group.id to
val uniqueGroupId =
s"spark-kafka-source-${UUID.randomUUID}-${metadataPath.hashCode}"
see, i.e.,
https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L124
I guess Spark developers had a good reason to fix it, but is it possible to
make configurable the prefix of the above uniqueGroupId ("spark-kafka-source")?
If so, I could prepare a PR on it.
The rational is that we do not want all spark-jobs to use the same certificate
on group-ids of the form (spark-kafka-source-*).
Best regards,Anastasios Zouzias
