You should be able to override the Kafka client version. The Kafka APIs used by Structured Streaming exist in new Kafka versions. There is a known correctness issue <https://issues.apache.org/jira/browse/KAFKA-4547> in Kafka 0.10.1.*. Other versions should be fine.
Best Regards, Ryan On Tue, Apr 28, 2020 at 1:54 PM Ahn, Daniel <daniel....@optum.com.invalid> wrote: > I have a keberized HDFS cluster. When I use structured streaming with > Kafka (with SASL_SSL/PLAINTEXT), I believe I’m blocked by Kafka-5294. > > > > It seems like fix version in 0.11.0.0 Kafka client library. I have a Spark > 2.3 cluster, and it’s using 0.10.0.1 kafka client library. Do you know if I > can override kafka client version during runtime? > > > > - Dan > > > > > > > This e-mail, including attachments, may include confidential and/or > proprietary information, and may be used only by the person or entity > to which it is addressed. If the reader of this e-mail is not the intended > recipient or his or her authorized agent, the reader is hereby notified > that any dissemination, distribution or copying of this e-mail is > prohibited. If you have received this e-mail in error, please notify the > sender by replying to this message and delete this e-mail immediately. >
