Scott, I want to apologize, I took entirely too long to come up with a design doc for us all to discuss. If you still have interest in adding the Heron Simulator to ECO please review this doc and make comments as needed. It's not meant to be "this is how it's going to be done", it's more of a conversation starter to get us all on the same page of what needs to be done.
https://docs.google.com/document/d/1UpgwWSIiJ_GBfU-ZAN0b8sVuiVNVutpRv_1oSGhynmE/edit?usp=sharing If anyone else has any questions, comments, or concerns, please let me know. - Josh On Wed, Jun 9, 2021 at 8:01 AM Scott Mayne <[email protected]> wrote: > Thanks Josh, sounds good. > > Apologies for my over-describing, just wanted to make sure I was being > clear. > > Scott > > ------------------------------ > *From:* Josh Fischer <[email protected]> > *Sent:* June 8, 2021 2:57 PM > *To:* [email protected] <[email protected]> > *Cc:* [email protected] <[email protected]>; > [email protected] <[email protected]> > *Subject:* Re: ECO + YAML in IDE > > Got it. My thinking out loud via email doesn't always work out so well. > I'll come up with a design doc to share that we can all talk about and > hopefully clarify things. Give me a few days and I'll get it out to the > list. > > Hope that works for you, > > - Josh > > On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <[email protected]> > wrote: > > Hi Josh, > > My first reaction is for it to not be specified in the topology YAML > (seems easier to handle if nothing changes in that file, however deployed). > > But it's certainly not a dealbreaker and will defer to your opinion as to > what's best from an overall Heron point of view. > > This new Simulator functionality in EcoSubmitter, it will allow an > environment-specific HeronConfig file to be passed in? > > And the Flux syntax didn't need a JAR path with the "local" invocation, > just to reiterate the goal here is to have Eco use the IDE workspace files > (Java files for spouts, bolts, etc.) to allow breakpoints, etc. > > Thanks, > Scott > > ------------------------------ > *From:* Josh Fischer <[email protected]> > *Sent:* June 6, 2021 10:10 PM > *To:* [email protected] <[email protected]>; [email protected] < > [email protected]>; [email protected] < > [email protected]> > *Subject:* Re: ECO + YAML in IDE > > Hi Scott, > > I think we would need to make changes to the EcoSubmitter class to allow > for use of the Simulator. How would you prefer to specify that topology > would be run in the simulator? Would you prefer to specify this in the > topology yaml file or would you rather specify it as a command line > argument, something like > """ > heron submit local \ > some/path/to/jar.jar \ > org.apache.heron.eco.Eco \ > --eco path/to/topology.yaml --simulate > """ > If we pass --simulate as a parameter for the Eco deployment I have a > concern that it could be confusing for the other deployment methods since > it would be ignored for something like the Java Topology API as we would > specify this in the java code, not a deployment parameter. > > > - Josh > > > > On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <[email protected]> wrote: > > Hi Scott, > > > I’ll have time to look in to this tomorrow for you. I’ve also sent this to > the dev@heron list in case someone else would be interested. > > - Josh > > > On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <[email protected]> > wrote: > > Hi all, > > I asked a question about running ECO in local mode a while back, thanks to > Josh for suggesting the verbose flag but nothing was added to the console > that shed any light on why it exited right away. > > In the meantime, I got things running in IntelliJ > using org.apache.heron.simulator.Simulator, after creating the topology > using org.apache.heron.api.topology.TopologyBuilder. > > Now that that's working, we've circled back to the ECO/IDE/YAML question. > We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for > cluster deploys, and in an IDE with a main() method using Flux as well (see > bottom of message for my unsuccessful attempt to mimic the Flux way with > Eco). > > Obviously Storm 1.0.6 is getting a little long in the tooth and we're > exploring moving to Heron (and liking what we see so far). > > A nice thing about the Storm setup was we use one YAML file to define the > topologies in both scenarios. So far in Heron, it would be ECO/YAML for > cluster (not that we've gotten that far yet) and TopologyBuilder in the > IDE for development. > > Would be nice to not have this difference and use YAML for both (avoid > mismatches between the two, etc.), so my question is: Has anyone set up > Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology > from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If > so, how? > > As background, these Storm ETLs were written before I joined the team, and > I'm relatively new to streaming frameworks, so I might be missing something > obvious here. > > For example, I just removed the Storm artifacts from the POM, added Heron, > and modified the ETL code as needed. Haven't installed Heron binaries or > anything like that on my laptop, not sure if that's a prerequisite. > > Thanks, > Scott > > public class KafkaTopology { > > public static void main(String[] args) throws Exception { > > List<String> argList = new ArrayList<String>(Arrays.asList(args)); > String file = > KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile(); > argList.add("local"); > argList.add("--eco-config-file"); > argList.add(file); > > file = > KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile(); > argList.add("--props"); > argList.add(file); > > argList.add("--sleep"); // something from Flux to make it run for a > while > argList.add("36000000"); > > String[] ecoArgs = argList.toArray(new String[argList.size()]); > > Eco.main(ecoArgs); > > } > } > > > -- > Sent from A Mobile Device > >
