Hi Spark team, We encountered a `NoSuchMethodError` when running a PySpark application with Spark 4.0.0 and kafka-clients-4.0.0:
java.lang.NoSuchMethodError: org.apache.kafka.clients.admin.DescribeTopicsResult.all() This appears to be due to Spark’s Kafka integration module still calling `.all()`, which was removed in Kafka 4.0.0 (see KAFKA-18289). Downgrading to kafka-clients-3.6.1 resolves the issue. Is this a known compatibility gap? Should Spark 4.0.0 be updated to use `allTopicNames()` instead? Thanks! -- Sandeep Ballu Software Engineer Consultant Team: Datasync & Datalake sba...@tsys.com globalpayments.com -- NOTICE: This email message is for the sole use of the addressee(s) named above and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this message or any attachments is expressly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies and backups of the original message.
