Yeah, that's not it. The stateful scaling part of that doc got lengthy enough that we broke it into a doc of it's own, per Sanjeev's suggestion IIRC. The fact that I can't locate it makes me think it was a twitter doc of mine (although it was not Twitter-specific), which I'm sure was shared with Sanjeev, Maosong and probably Karthik. If you can find it, please share.
On Mon, May 7, 2018 at 12:41 AM, Ning Wang <wangnin...@gmail.com> wrote: > Thanks Karthik. The doc is not exactly the same but close enough. > > It seems my doc is an internal one so let's use your doc as reference. I > will see if there is any major differences and comment. > > On Sun, May 6, 2018 at 1:22 PM, Karthik Ramasamy <kart...@streaml.io> > wrote: > >> Here it is >> >> https://docs.google.com/document/d/1YDFNvLTX6Sg3WDrNFKiWLaJv >> uEtK4eyxEaA0w9cVlG4/edit#heading=h.d6uy2uxfs2xq >> >> cheers >> /karthik >> >> >> On Sun, May 6, 2018 at 8:20 AM, Bill Graham <billgra...@gmail.com> wrote: >> >>> Can you share the doc please? >>> >>> On Sat, May 5, 2018 at 4:18 PM Ning Wang <wangnin...@gmail.com> wrote: >>> >>> > Thanks. >>> > >>> > Yeah I have read the design doc. It has a section for scaling and >>> covers >>> > some designs but not reaching this level of details I am afraid. >>> > >>> > On Sat, May 5, 2018 at 9:45 AM, Bill Graham <billgra...@gmail.com> >>> wrote: >>> > >>> >> The stateful processing design included a large section on scaling, >>> which >>> >> was intended to be done as a future phase. It's very similar to what's >>> >> being described. Sanjeev and I worked on it about a 1.5 years ago with >>> >> Maosong and it was in a google doc. Sanjeev do you have that design >>> doc? I >>> >> can't seem locate it. >>> >> >>> >> On Sat, May 5, 2018 at 12:03 AM, Ning Wang <wangnin...@gmail.com> >>> wrote: >>> >> >>> >> > If we go this way, we need key -> state map for each component so >>> that >>> >> the >>> >> > state data can be repartitioned. >>> >> > >>> >> > On Fri, May 4, 2018 at 11:44 PM, Karthik Ramasamy < >>> kart...@streaml.io> >>> >> > wrote: >>> >> > >>> >> > > Instead - if it references >>> >> > > >>> >> > > topology name + component name + key range >>> >> > > >>> >> > > will it be better? >>> >> > > >>> >> > > cheers >>> >> > > /karthik >>> >> > > >>> >> > > >>> >> > > On Fri, May 4, 2018 at 11:23 PM, Ning Wang <wangnin...@gmail.com> >>> >> wrote: >>> >> > > >>> >> > > > Currently I think each Instance serializes the state object >>> into a >>> >> byte >>> >> > > > array and checkpoint manager saves the byte array into a file. >>> The >>> >> file >>> >> > > is >>> >> > > > referenced by topology name + component name + instance id. >>> >> > > > >>> >> > > > On Fri, May 4, 2018 at 11:10 PM, Karthik Ramasamy < >>> >> kart...@streaml.io> >>> >> > > > wrote: >>> >> > > > >>> >> > > > > I am not sure I understand why the state is tied to an >>> instance? >>> >> > > > > >>> >> > > > > cheers >>> >> > > > > /karthik >>> >> > > > > >>> >> > > > > On Fri, May 4, 2018 at 4:36 PM, Thomas Cooper < >>> >> > tom.n.coo...@gmail.com> >>> >> > > > > wrote: >>> >> > > > > >>> >> > > > > > Yeah, state recovery is a bit more difficult with Heron's >>> >> > > architecture. >>> >> > > > > In >>> >> > > > > > Storm, the task IDs are not just values used for routing >>> they >>> >> > > actually >>> >> > > > > > equate to a task instance within the executor. An executor >>> which >>> >> > > > > currently >>> >> > > > > > processes the keys 4-8 actually contains 5 task instances >>> of the >>> >> > same >>> >> > > > > > component. So for each task, they just save its state >>> attached >>> >> to >>> >> > the >>> >> > > > > > single task ID and reassemble executors with the new task >>> >> > instances. >>> >> > > > > > >>> >> > > > > > We don't want or have to do that with Heron instances but we >>> >> would >>> >> > > need >>> >> > > > > to >>> >> > > > > > have some way to have a state change tied to the task (or >>> >> routing >>> >> > key >>> >> > > > if >>> >> > > > > we >>> >> > > > > > go to the key range idea). For something like a word count >>> you >>> >> > might >>> >> > > > save >>> >> > > > > > counts using a nested map like: { routing key : {word : >>> count >>> >> }}. >>> >> > The >>> >> > > > > > routing key could be included in the Tuple instance. >>> However, >>> >> > whether >>> >> > > > > this >>> >> > > > > > pattern would work for more generic state cases I don't >>> know? >>> >> > > > > > >>> >> > > > > > Tom Cooper >>> >> > > > > > W: www.tomcooper.org.uk | Twitter: @tomncooper >>> >> > > > > > <https://twitter.com/tomncooper> >>> >> > > > > > >>> >> > > > > > >>> >> > > > > > On Fri, 4 May 2018 at 15:54, Neng Lu <freen...@gmail.com> >>> >> wrote: >>> >> > > > > > >>> >> > > > > > > +1 for this idea. As long as the predefined key space is >>> large >>> >> > > > enough, >>> >> > > > > it >>> >> > > > > > > should work for most of the cases. >>> >> > > > > > > >>> >> > > > > > > Based on my experience with topologies, I never saw one >>> >> component >>> >> > > has >>> >> > > > > > more >>> >> > > > > > > than 1000 instances in a topology. >>> >> > > > > > > >>> >> > > > > > > For recovering states from an update, there will be some >>> >> problems >>> >> > > > > though. >>> >> > > > > > > Since the states stored in heron are strongly connected >>> with >>> >> each >>> >> > > > > > instance, >>> >> > > > > > > we either need to have >>> >> > > > > > > some resolver does the state repartitioning or stores >>> states >>> >> with >>> >> > > the >>> >> > > > > key >>> >> > > > > > > instead of with each instance. >>> >> > > > > > > >>> >> > > > > > > >>> >> > > > > > > >>> >> > > > > > > On Fri, May 4, 2018 at 3:01 PM, Karthik Ramasamy < >>> >> > > > kramas...@gmail.com> >>> >> > > > > > > wrote: >>> >> > > > > > > >>> >> > > > > > > > Thanks for sharing. I like the Storm approach >>> >> > > > > > > > >>> >> > > > > > > > - keeps the implementation simpler >>> >> > > > > > > > - state is deterministic across restarts >>> >> > > > > > > > - makes it easy to reason and debug >>> >> > > > > > > > >>> >> > > > > > > > The hard limit is not a problem at all since most of the >>> >> > > topologies >>> >> > > > > > will >>> >> > > > > > > > be never that big. >>> >> > > > > > > > If you can handle Twitter topologies cleanly, it is more >>> >> that >>> >> > > > > > sufficient >>> >> > > > > > > I >>> >> > > > > > > > believe. >>> >> > > > > > > > >>> >> > > > > > > > cheers >>> >> > > > > > > > /karthik >>> >> > > > > > > > >>> >> > > > > > > > > On May 4, 2018, at 2:31 PM, Thomas Cooper < >>> >> > > > tom.n.coo...@gmail.com> >>> >> > > > > > > > wrote: >>> >> > > > > > > > > >>> >> > > > > > > > > Hi all, >>> >> > > > > > > > > >>> >> > > > > > > > > A while ago I emailed about the issue of how fields >>> (key) >>> >> > > grouped >>> >> > > > > > > routing >>> >> > > > > > > > > in Heron was not consistent across an update and how >>> this >>> >> > makes >>> >> > > > > > > > preserving >>> >> > > > > > > > > state across an update very difficult and also makes >>> it >>> >> > > > > > > > > difficult/impossible to analyse or predict tuple flows >>> >> > through >>> >> > > a >>> >> > > > > > > > > current/proposed topology physical plan. >>> >> > > > > > > > > >>> >> > > > > > > > > I suggested adopting Storms approach of pre-defining a >>> >> > routing >>> >> > > > key >>> >> > > > > > > > > space for each component (eg 0-999), so that instead >>> of an >>> >> > > > instance >>> >> > > > > > > > having >>> >> > > > > > > > > a single task id that gets reset at every update (eg >>> 10) >>> >> it >>> >> > > has a >>> >> > > > > > range >>> >> > > > > > > > of >>> >> > > > > > > > > id's (eg 10-16) that changes depending on the >>> parallelism >>> >> of >>> >> > > the >>> >> > > > > > > > component. >>> >> > > > > > > > > This has the advantage that a key will always hash to >>> the >>> >> > same >>> >> > > > task >>> >> > > > > > ID >>> >> > > > > > > > for >>> >> > > > > > > > > the lifetime of the topology. Meaning recovering state >>> >> for an >>> >> > > > > > instance >>> >> > > > > > > > > after a crash or update is just a case of pulling the >>> >> state >>> >> > > > linked >>> >> > > > > to >>> >> > > > > > > the >>> >> > > > > > > > > keys in its task ID range. >>> >> > > > > > > > > >>> >> > > > > > > > > I know the above proposal has issues, not least of all >>> >> > placing >>> >> > > a >>> >> > > > > hard >>> >> > > > > > > > upper >>> >> > > > > > > > > limit on the scale out of a component, and that some >>> >> > > alternative >>> >> > > > > > ideas >>> >> > > > > > > > are >>> >> > > > > > > > > being floated for solving the stateful update issue. >>> >> > However, I >>> >> > > > > just >>> >> > > > > > > > wanted >>> >> > > > > > > > > to throw some more weight behind the Storm approach. >>> There >>> >> > was >>> >> > > a >>> >> > > > > > recent >>> >> > > > > > > > > paper about high-performance network load balancing >>> >> > > > > > > > > <https://blog.acolyer.org/2018/05/03/stateless- >>> >> > > > > > > > datacenter-load-balancing-with-beamer/>that >>> >> > > > > > > > > describes an approach using a fixed key space similar >>> to >>> >> > > Storm's >>> >> > > > > (see >>> >> > > > > > > the >>> >> > > > > > > > > section called Stable Hashing - they assign a range >>> 100x >>> >> the >>> >> > > > > expected >>> >> > > > > > > > > connection pool size - which we could do with heron to >>> >> > prevent >>> >> > > > ever >>> >> > > > > > > > hitting >>> >> > > > > > > > > the upper scaling limit). Also, this new load >>> balancer, >>> >> > Beamer, >>> >> > > > > > claims >>> >> > > > > > > to >>> >> > > > > > > > > be twice as fast as Google's Maglev >>> >> > > > > > > > > <https://blog.acolyer.org/2016 >>> /03/21/maglev-a-fast-and- >>> >> > > > > > > > reliable-software-network-load-balancer/> >>> >> > > > > > > > > which again uses a pre-defined keyspace and ID ranges >>> to >>> >> > create >>> >> > > > > > look-up >>> >> > > > > > > > > tables deterministically. >>> >> > > > > > > > > >>> >> > > > > > > > > I know a load balancer is a different beast to a >>> stream >>> >> > > grouping >>> >> > > > > but >>> >> > > > > > > > there >>> >> > > > > > > > > are some interesting ideas in those papers (The links >>> >> point >>> >> > to >>> >> > > > > > summary >>> >> > > > > > > > blog >>> >> > > > > > > > > posts so you don't have to read the whole paper). >>> >> > > > > > > > > >>> >> > > > > > > > > Anyway, I just thought I would those papers out there >>> and >>> >> see >>> >> > > > what >>> >> > > > > > > people >>> >> > > > > > > > > think. >>> >> > > > > > > > > >>> >> > > > > > > > > Tom Cooper >>> >> > > > > > > > > W: www.tomcooper.org.uk | Twitter: @tomncooper >>> >> > > > > > > > > <https://twitter.com/tomncooper> >>> >> > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > >>> >> > > > > > >>> >> > > > > >>> >> > > > >>> >> > > >>> >> > >>> >> >>> > >>> > -- >>> Sent from Gmail Mobile >>> >> >> >
