Hi, I prepared this notes doc trying to capture the ideas and thoughts we have so far.
https://docs.google.com/document/d/1tY2rBKw9c-WBGde8oWKlIq0Owvhqo04tm1hbFQpLuQ4/edit# It is very likely that I missed a lot of things, plus it is still early stage now and brainstorming can be super helpful. So please feel free to add/comment/edit. Thanks in advance! On Fri, Nov 17, 2017 at 5:22 PM, Fu Maosong <[email protected]> wrote: > @Sanjeev and @ning could provide more details on them, like pros&cons. > > 2017-11-17 17:11 GMT-08:00 Saikat Kanjilal <[email protected]>: > >> Hi Fu, >> Thanks for the proposals, is it possible to outline the pros and cons >> that you guys discussed? That would help the community in weighing in on >> this. >> Thanks in advance >> >> Sent from my iPhone >> >> On Nov 17, 2017, at 4:15 PM, Fu Maosong <[email protected]> wrote: >> >> Today, Ning, Huijun, Sanjeev and Maosong discussed about the >> bp(back-pressure) today, and here are the notes: >> >> *Two major issues of current bp algorithm*: >> 1. one single point of bp will stop the whole topology >> 2. when a topology in bp state, the overall throughput for the whole >> topology can reduce >> >> *Some proposals*: >> For 1. >> - load shedding -- for example, drop tuples in stmgr. >> - better bp algo. -- no need to stop the whole topology; can just handle >> the slowness to particular instances, for example, spill the buffer to the >> disk >> >> For 2. >> - Rate control on the source(spout) side. So even in bp state, the overall >> throughput can be higher than normal >> - Run-time scale-up >> >> Let me know if I missing anything. >> >> -- >> With my best Regards >> ------------------ >> Fu Maosong >> Twitter Inc. >> Mobile: +001-415-244-7520 >> >> > > > -- > With my best Regards > ------------------ > Fu Maosong > Twitter Inc. > Mobile: +001-415-244-7520 >
