HyukjinKwon opened a new pull request, #58613: URL: https://github.com/apache/spark/pull/58613
### What changes were proposed in this pull request? Add an opt-in, internal `spark.sql.kafka.disallowedOptions` (default empty). When set to a comma-separated list of Kafka client option names (without the `kafka.` prefix), those options are rejected if passed as `kafka.<name>` on the Kafka source or sink; an empty list (the default) allows all options and preserves the previous behavior. The rejection is raised through the Kafka connector's error-class framework (a new `KAFKA_DISALLOWED_OPTION` condition). This is one of three PRs split out of a combined data-source-options change; the Avro schema-URL and Hive class-loading parts are handled separately. ### Why are the changes needed? Gives operators an optional control over which Kafka client options an application may set through the data source. It defaults to empty, so there is no behavior change unless configured. ### Does this PR introduce _any_ user-facing change? No by default. When `spark.sql.kafka.disallowedOptions` is set, a listed `kafka.<name>` option on a Kafka read or write is rejected with a clear error. ### How was this patch tested? New `KafkaSourceProviderSuite` cases: a listed option is rejected on both the source and the sink path, and the empty default (unset) still accepts the option. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 This pull request and its description were written by Isaac. Co-authored-by: Isaac <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
