Hi, You can set spark.streaming.kafka.backpressure.enable=true. If your tasks can't process larger data that this variable can control the kafka data into streaming speed. And you can increment your streaming process time window.
Sent from my Samsung Galaxy smartphone. -------- Original message -------- From: zenglong chen <[email protected]> Date: 8/2/19 09:59 (GMT+08:00) To: [email protected] Subject: spark stream kafka wait for all data process done How can kafka wait for tasks process done then begin receive next batch?I want to process 5000 record once by pandas and it may take too long time to process.
