Re: spark stream kafka wait for all data process done

[刘 勇]

Hi,
You can set spark.streaming.kafka.backpressure.enable=true.
If your tasks can't process larger data that this variable can control the 
kafka data into streaming speed. And you can increment your streaming process 
time window.



Sent from my Samsung Galaxy smartphone.


 Original message 
From: zenglong chen 
Date: 8/2/19 09:59 (GMT+08:00)
To: user@spark.apache.org
Subject: spark stream kafka wait for all data process done

How can kafka wait for tasks process done then begin receive next batch?I want 
to process 5000 record once by pandas and it may take too long time to process.

spark stream kafka wait for all data process done

[zenglong chen]

How can kafka wait for tasks process done then begin receive next batch?I
want to process 5000 record once by pandas and it may take too long time to
process.