Adding to Gabor's answer, in Spark 3.0 end users can even provide full of group id (Please refer SPARK-26350 [1]), but you may feel more convenient to use prefix of group id Gabor guided (Please refer SPARK-26121 [2]) to provide permission to broader ranges of groups.
1. https://issues.apache.org/jira/browse/SPARK-26350 2. https://issues.apache.org/jira/browse/SPARK-26121 Thanks, Jungtaek Lim (HeartSaVioR) 2019년 2월 13일 (수) 오후 6:36, Gabor Somogyi <gabor.g.somo...@gmail.com>님이 작성: > Hi Thomas, > > The issue occurs when the user does not have the READ permission on the > consumer groups. > > In DStreams group ID is configured in application, for example: > https://github.com/gaborgsomogyi/spark-dstream-secure-kafka-app/blob/161bf02eb3677aac604d63499041f72231d0e371/src/main/scala/com/cloudera/spark/examples/DirectKafkaWordCount.scala#L59 > > In Strucuted Streaming the group ID is generated by Spark internally. > > Either one has to give access to "spark-kafka-source-*" group or in Spark > 3.0 this prefix can be configured with "groupidprefix" parameter. > > BR, > G > > > On Wed, Feb 13, 2019 at 3:58 AM Allu👌🏽 Thomas > <thomaspt...@icloud.com.invalid> wrote: > >> Hi There, >> >> My use case is to read a simple json message from Kafka queue using Spark >> Structured Streaming. But I’m getting the following error message when I >> run my Kafka consumer. I don’t get this error when using Spark direct >> stream. The issue is happening only with structured streaming. Any help >> would be greatly appreciated. >> >> >> Exception in thread "main" >> org.apache.spark.sql.streaming.StreamingQueryException: Not authorized to >> access group: >> spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2 >> === Streaming Query === >> Identifier: [id = 6ab10eab-4f71-435c-8705-820e66cee47e, runId = >> 48430367-9e14-450b-b8e0-27199b536403] >> Current Committed Offsets: {} >> Current Available Offsets: {} >> >> >> Current State: ACTIVE >> Thread State: RUNNABLE >> >> >> Logical Plan: >> KafkaSource[Subscribe[cla-claim-raw]] >> at org.apache.spark.sql.execution.streaming.StreamExecution.org >> $apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295) >> at >> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189) >> Caused by: org.apache.kafka.common.errors.GroupAuthorizationException: >> Not authorized to access group: >> spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2 >> >> Thanks, >> Thomas Thomas >> >
