[GitHub] kramasamy opened a new pull request #2893: pull yaml cpp for official release page

kramasamy opened a new pull request #2893: pull yaml cpp for official release page URL: https://github.com/apache/incubator-heron/pull/2893 This is an automated message from the Apache Git Service. To respond to the

Re: Stateful updating and deterministic routing

Thanks. Yeah I have read the design doc. It has a section for scaling and covers some designs but not reaching this level of details I am afraid. On Sat, May 5, 2018 at 9:45 AM, Bill Graham wrote: > The stateful processing design included a large section on scaling, which

[GitHub] joshfischer1108 commented on issue #2892: fixing eco parser test

joshfischer1108 commented on issue #2892: fixing eco parser test URL: https://github.com/apache/incubator-heron/pull/2892#issuecomment-386822849 @kramasamy Once master build is fixed I will merge in those changes This is an

Re: Stateful updating and deterministic routing

The stateful processing design included a large section on scaling, which was intended to be done as a future phase. It's very similar to what's being described. Sanjeev and I worked on it about a 1.5 years ago with Maosong and it was in a google doc. Sanjeev do you have that design doc? I can't

[GitHub] joshfischer1108 commented on a change in pull request #2892: fixing eco parser test

joshfischer1108 commented on a change in pull request #2892: fixing eco parser test URL: https://github.com/apache/incubator-heron/pull/2892#discussion_r186265179 ## File path: eco/tests/java/org/apache/heron/eco/parser/EcoParserTest.java ## @@ -33,17 +140,17 @@ +

Re: Stateful updating and deterministic routing

If we go this way, we need key -> state map for each component so that the state data can be repartitioned. On Fri, May 4, 2018 at 11:44 PM, Karthik Ramasamy wrote: > Instead - if it references > > topology name + component name + key range > > will it be better? > > cheers

[GitHub] srkukarni commented on a change in pull request #2891: [WIP] Refactor StatefulStorage

srkukarni commented on a change in pull request #2891: [WIP] Refactor StatefulStorage URL: https://github.com/apache/incubator-heron/pull/2891#discussion_r186254708 ## File path: heron/proto/ckptmgr.proto ## @@ -171,7 +175,25 @@ message CleanStatefulCheckpointResponse {

Re: Stateful updating and deterministic routing

Instead - if it references topology name + component name + key range will it be better? cheers /karthik On Fri, May 4, 2018 at 11:23 PM, Ning Wang wrote: > Currently I think each Instance serializes the state object into a byte > array and checkpoint manager saves the

Re: Stateful updating and deterministic routing

Currently I think each Instance serializes the state object into a byte array and checkpoint manager saves the byte array into a file. The file is referenced by topology name + component name + instance id. On Fri, May 4, 2018 at 11:10 PM, Karthik Ramasamy wrote: > I am not

Re: Stateful updating and deterministic routing

I am not sure I understand why the state is tied to an instance? cheers /karthik On Fri, May 4, 2018 at 4:36 PM, Thomas Cooper wrote: > Yeah, state recovery is a bit more difficult with Heron's architecture. In > Storm, the task IDs are not just values used for routing