Re: Back pressure not working on streaming

[Cody Koeninger]

That article is pretty old, If you click through the link to the jira
mentioned in it, https://issues.apache.org/jira/browse/SPARK-18580 ,
it's been resolved.

On Wed, Jan 2, 2019 at 12:42 AM JF Chen  wrote:
>
> yes, 10 is a very low value for testing initial rate.
> And from this article 
> https://www.linkedin.com/pulse/enable-back-pressure-make-your-spark-streaming-production-lan-jiang/,
>  it seems spark back pressure is not available for dstream?
> So ,max rate per partition is the only available back pressure solution for 
> kafka dstream input?
>
> Regard,
> Junfeng Chen
>
>
> On Wed, Jan 2, 2019 at 11:49 AM HARSH TAKKAR  wrote:
>>
>> There is separate property for max rate , by default is is not set, so if 
>> you want to limit the max rate you should  provide that property  a value.
>>
>> Initial rate =10 means it will pick only 10 records per receiver in the 
>> batch interval when you start the process.
>>
>> Depending  upon the consumption rate it will increase  the consumption of 
>> records for processing in each batch.
>>
>> However i, feel 10 is way to low number for 32 partitioned kafka topic.
>>
>>
>>
>> Regards
>> Harsh
>> Happy New Year
>>
>> On Wed 2 Jan, 2019, 08:33 JF Chen >>
>>> I have set  spark.streaming.backpressure.enabled to true,  
>>> spark.streaming.backpressure.initialRate to 10.
>>> Once my application started, it received 32 million messages on first batch.
>>> My application runs every 300 seconds, with 32 kafka partition. So what's 
>>> is the max rate if I set initial rate to 10?
>>>
>>> Thanks!
>>>
>>>
>>> Regard,
>>> Junfeng Chen

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Back pressure not working on streaming

[JF Chen]

yes, 10 is a very low value for testing initial rate.
And from this article
https://www.linkedin.com/pulse/enable-back-pressure-make-your-spark-streaming-production-lan-jiang/,
it seems spark back pressure is not available for dstream?
So ,max rate per partition is the only available back pressure solution for
kafka dstream input?

Regard,
Junfeng Chen


On Wed, Jan 2, 2019 at 11:49 AM HARSH TAKKAR  wrote:

> There is separate property for max rate , by default is is not set, so if
> you want to limit the max rate you should  provide that property  a value.
>
> Initial rate =10 means it will pick only 10 records per receiver in the
> batch interval when you start the process.
>
> Depending  upon the consumption rate it will increase  the consumption of
> records for processing in each batch.
>
> However i, feel 10 is way to low number for 32 partitioned kafka topic.
>
>
>
> Regards
> Harsh
> Happy New Year
>
> On Wed 2 Jan, 2019, 08:33 JF Chen 
>> I have set  spark.streaming.backpressure.enabled to true,  
>> spark.streaming.backpressure.initialRate
>> to 10.
>> Once my application started, it received 32 million messages on first
>> batch.
>> My application runs every 300 seconds, with 32 kafka partition. So what's
>> is the max rate if I set initial rate to 10?
>>
>> Thanks!
>>
>>
>> Regard,
>> Junfeng Chen
>>
>

Re: Back pressure not working on streaming

[Dillon Bostwick]

Unsubscribe

On Tue, Jan 1, 2019 at 10:03 PM JF Chen  wrote:

> I have set  spark.streaming.backpressure.enabled to true,  
> spark.streaming.backpressure.initialRate
> to 10.
> Once my application started, it received 32 million messages on first
> batch.
> My application runs every 300 seconds, with 32 kafka partition. So what's
> is the max rate if I set initial rate to 10?
>
> Thanks!
>
>
> Regard,
> Junfeng Chen
>
-- 

Dillon Bostwick
Solutions Engineer
Databricks 
678-770-5344

Re: Back pressure not working on streaming

[HARSH TAKKAR]

There is separate property for max rate , by default is is not set, so if
you want to limit the max rate you should  provide that property  a value.

Initial rate =10 means it will pick only 10 records per receiver in the
batch interval when you start the process.

Depending  upon the consumption rate it will increase  the consumption of
records for processing in each batch.

However i, feel 10 is way to low number for 32 partitioned kafka topic.



Regards
Harsh
Happy New Year

On Wed 2 Jan, 2019, 08:33 JF Chen  I have set  spark.streaming.backpressure.enabled to true,  
> spark.streaming.backpressure.initialRate
> to 10.
> Once my application started, it received 32 million messages on first
> batch.
> My application runs every 300 seconds, with 32 kafka partition. So what's
> is the max rate if I set initial rate to 10?
>
> Thanks!
>
>
> Regard,
> Junfeng Chen
>

Back pressure not working on streaming

[JF Chen]

I have set  spark.streaming.backpressure.enabled to true,
spark.streaming.backpressure.initialRate
to 10.
Once my application started, it received 32 million messages on first
batch.
My application runs every 300 seconds, with 32 kafka partition. So what's
is the max rate if I set initial rate to 10?

Thanks!


Regard,
Junfeng Chen