viirya commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r499874400
##########
File path: docs/ss-migration-guide.md
##########
@@ -26,6 +26,19 @@ Note that this migration guide describes the items specific
to Structured Stream
Many items of SQL migration can be applied when migrating Structured Streaming
to higher versions.
Please refer [Migration Guide: SQL, Datasets and
DataFrame](sql-migration-guide.html).
+## Upgrading from Structured Streaming 3.0 to 3.1
+
+- In Spark 3.0 and below, secure Kafka processing needed the following ACLs
from driver perspective:
+ * Topic resource describe operation
+ * Topic resource read operation
+ * Group resource read operation
+
+ Since Spark 3.1, offsets are obtained with `AdminClient` instead of
`KafkaConsumer` and now the following ACLs needed from driver perspective:
+ * Topic resource describe operation
+
+ Since `AdminClient` in driver is not connecting to consumer group,
`group.id` based authorization will not work anymore (executors never done
group based authorization).
Review comment:
What it means `executors never done group based authorization`? Seems on
executors, we use `group.id` if it is set by users, no?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
