Hi Todd,
Thanks a lot, that works. Althouhg I am curious whether you know why the
initialRate setting not kicking in?
But for now the pipeline is usable again. Thanks a lot.
Thanks & Regards
Biplob Biswas
On Thu, Jul 26, 2018 at 3:03 PM Todd Nist wrote:
> Have you tried reducing the
Have you tried reducing the maxRatePerPartition to a lower value? Based on
your settings, I believe you are going to be able to pull *600K* worth of
messages from Kafka, basically:
• maxRatePerPartition=15000
• batchInterval 10s
• 4 partitions on Ingest topic
This results in a maximum
Hi Todd,
Thanks for the reply. I have the mayxRatePerPartition set as well. Below is
the spark submit config we used and still got the issue. Also the *batch
interval is set at 10s* and *number of partitions on the topic is set to 4*
:
spark2-submit --name "${YARN_NAME}" \
--master yarn \
Hi Biplob,
How many partitions are on the topic you are reading from and have you set
the maxRatePerPartition? iirc, spark back pressure is calculated as
follows:
*Spark back pressure:*
Back pressure is calculated off of the following:
• maxRatePerPartition=200
• batchInterval 30s
• 3
Did anyone face similar issue? and any viable way to solve this?
Thanks & Regards
Biplob Biswas
On Wed, Jul 25, 2018 at 4:23 PM Biplob Biswas
wrote:
> I have enabled the spark.streaming.backpressure.enabled setting and also
> set spark.streaming.backpressure.initialRate to 15000, but my
