I think you should be using a foreachPartition and a broadcast to build your producer. From there you will have full control of all options and serialization needed via direct access to the KafkaProducer, as well as all options therein associated (e.g. callbacks, interceptors, etc).
-dan On Mon, Feb 24, 2025 at 6:26 AM Abhishek Singla <abhisheksingla...@gmail.com> wrote: > Hi Team, > > I am using spark to read from S3 and write to Kafka. > > Spark Version: 3.1.2 > Scala Version: 2.12 > Spark Kafka connector: spark-sql-kafka-0-10_2.12 > > I want to throttle kafka producer. I tried using *linger.ms > <http://linger.ms>* and *batch.size* config but I can see in *ProducerConfig: > ProducerConfig values* at runtime that they are not being set. Is there > something I am missing? Is there any other way to throttle kafka writes? > > *dataset.write().format("kafka").options(options).save();* > > Regards, > Abhishek Singla > > > > >
