Re: Notes on bp(back-pressure) discussion on Nov.17

[Saikat Kanjilal]

Hi Fu,
Thanks for the proposals, is it possible to outline the pros and cons that you 
guys discussed?  That would help the community in weighing in on this.
Thanks in advance

Sent from my iPhone

On Nov 17, 2017, at 4:15 PM, Fu Maosong 
> wrote:

Today, Ning, Huijun, Sanjeev and Maosong discussed about the
bp(back-pressure) today, and here are the notes:

*Two major issues of current bp algorithm*:
1. one single point of bp will stop the whole topology
2. when a topology in bp state, the overall throughput for the whole
topology can reduce

*Some proposals*:
For 1.
- load shedding -- for example, drop tuples in stmgr.
- better bp algo. -- no need to stop the whole topology; can just handle
the slowness to particular instances, for example, spill the buffer to the
disk

For 2.
- Rate control on the source(spout) side. So even in bp state, the overall
throughput can be higher than normal
- Run-time scale-up

Let me know if I missing anything.

--
With my best Regards
--
Fu Maosong
Twitter Inc.
Mobile: +001-415-244-7520

Re: Release 0.17.1

[Ali Ahmed]

Lets process with the release.


> On Nov 17, 2017, at 4:21 PM, Jerry Peng  wrote:
> 
> sounds good
> 
> On Fri, Nov 17, 2017 at 4:06 PM, Karthik Ramasamy  wrote:
>> Sounds good!
>> 
>> On Fri, Nov 17, 2017 at 12:55 PM, Sanjeev Kulkarni 
>> wrote:
>> 
>>> Hi all,
>>> In light of the critical bug fix(https://github.com/
>>> twitter/heron/pull/2551),
>>> might make sense to make a new release today. Any objections?
>>> 

Re: Release 0.17.1

[Chris Kellogg]

sounds good

On Fri, Nov 17, 2017 at 4:21 PM, Jerry Peng 
wrote:

> sounds good
>
> On Fri, Nov 17, 2017 at 4:06 PM, Karthik Ramasamy 
> wrote:
> > Sounds good!
> >
> > On Fri, Nov 17, 2017 at 12:55 PM, Sanjeev Kulkarni 
> > wrote:
> >
> >> Hi all,
> >> In light of the critical bug fix(https://github.com/
> >> twitter/heron/pull/2551),
> >> might make sense to make a new release today. Any objections?
> >>
>

Re: Release 0.17.1

[Jerry Peng]

sounds good

On Fri, Nov 17, 2017 at 4:06 PM, Karthik Ramasamy  wrote:
> Sounds good!
>
> On Fri, Nov 17, 2017 at 12:55 PM, Sanjeev Kulkarni 
> wrote:
>
>> Hi all,
>> In light of the critical bug fix(https://github.com/
>> twitter/heron/pull/2551),
>> might make sense to make a new release today. Any objections?
>>

Notes on bp(back-pressure) discussion on Nov.17

[Fu Maosong]

Today, Ning, Huijun, Sanjeev and Maosong discussed about the
bp(back-pressure) today, and here are the notes:

*Two major issues of current bp algorithm*:
1. one single point of bp will stop the whole topology
2. when a topology in bp state, the overall throughput for the whole
topology can reduce

*Some proposals*:
For 1.
- load shedding -- for example, drop tuples in stmgr.
- better bp algo. -- no need to stop the whole topology; can just handle
the slowness to particular instances, for example, spill the buffer to the
disk

For 2.
- Rate control on the source(spout) side. So even in bp state, the overall
throughput can be higher than normal
- Run-time scale-up

Let me know if I missing anything.

-- 
With my best Regards
--
Fu Maosong
Twitter Inc.
Mobile: +001-415-244-7520

Re: Release 0.17.1

[Karthik Ramasamy]

Sounds good!

On Fri, Nov 17, 2017 at 12:55 PM, Sanjeev Kulkarni 
wrote:

> Hi all,
> In light of the critical bug fix(https://github.com/
> twitter/heron/pull/2551),
> might make sense to make a new release today. Any objections?
>

Release 0.17.1

[Sanjeev Kulkarni]

Hi all,
In light of the critical bug fix(https://github.com/twitter/heron/pull/2551),
might make sense to make a new release today. Any objections?