Got it. My thinking out loud via email doesn't always work out so well. I'll come up with a design doc to share that we can all talk about and hopefully clarify things. Give me a few days and I'll get it out to the list.
Hope that works for you, - Josh On Mon, Jun 7, 2021 at 10:57 AM Scott Mayne <[email protected]> wrote: > Hi Josh, > > My first reaction is for it to not be specified in the topology YAML > (seems easier to handle if nothing changes in that file, however deployed). > > But it's certainly not a dealbreaker and will defer to your opinion as to > what's best from an overall Heron point of view. > > This new Simulator functionality in EcoSubmitter, it will allow an > environment-specific HeronConfig file to be passed in? > > And the Flux syntax didn't need a JAR path with the "local" invocation, > just to reiterate the goal here is to have Eco use the IDE workspace files > (Java files for spouts, bolts, etc.) to allow breakpoints, etc. > > Thanks, > Scott > > ------------------------------ > *From:* Josh Fischer <[email protected]> > *Sent:* June 6, 2021 10:10 PM > *To:* [email protected] <[email protected]>; [email protected] < > [email protected]>; [email protected] < > [email protected]> > *Subject:* Re: ECO + YAML in IDE > > Hi Scott, > > I think we would need to make changes to the EcoSubmitter class to allow > for use of the Simulator. How would you prefer to specify that topology > would be run in the simulator? Would you prefer to specify this in the > topology yaml file or would you rather specify it as a command line > argument, something like > """ > heron submit local \ > some/path/to/jar.jar \ > org.apache.heron.eco.Eco \ > --eco path/to/topology.yaml --simulate > """ > If we pass --simulate as a parameter for the Eco deployment I have a > concern that it could be confusing for the other deployment methods since > it would be ignored for something like the Java Topology API as we would > specify this in the java code, not a deployment parameter. > > > - Josh > > > > On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <[email protected]> wrote: > > Hi Scott, > > > I’ll have time to look in to this tomorrow for you. I’ve also sent this to > the dev@heron list in case someone else would be interested. > > - Josh > > > On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <[email protected]> > wrote: > > Hi all, > > I asked a question about running ECO in local mode a while back, thanks to > Josh for suggesting the verbose flag but nothing was added to the console > that shed any light on why it exited right away. > > In the meantime, I got things running in IntelliJ > using org.apache.heron.simulator.Simulator, after creating the topology > using org.apache.heron.api.topology.TopologyBuilder. > > Now that that's working, we've circled back to the ECO/IDE/YAML question. > We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for > cluster deploys, and in an IDE with a main() method using Flux as well (see > bottom of message for my unsuccessful attempt to mimic the Flux way with > Eco). > > Obviously Storm 1.0.6 is getting a little long in the tooth and we're > exploring moving to Heron (and liking what we see so far). > > A nice thing about the Storm setup was we use one YAML file to define the > topologies in both scenarios. So far in Heron, it would be ECO/YAML for > cluster (not that we've gotten that far yet) and TopologyBuilder in the > IDE for development. > > Would be nice to not have this difference and use YAML for both (avoid > mismatches between the two, etc.), so my question is: Has anyone set up > Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology > from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If > so, how? > > As background, these Storm ETLs were written before I joined the team, and > I'm relatively new to streaming frameworks, so I might be missing something > obvious here. > > For example, I just removed the Storm artifacts from the POM, added Heron, > and modified the ETL code as needed. Haven't installed Heron binaries or > anything like that on my laptop, not sure if that's a prerequisite. > > Thanks, > Scott > > public class KafkaTopology { > > public static void main(String[] args) throws Exception { > > List<String> argList = new ArrayList<String>(Arrays.asList(args)); > String file = > KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile(); > argList.add("local"); > argList.add("--eco-config-file"); > argList.add(file); > > file = > KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile(); > argList.add("--props"); > argList.add(file); > > argList.add("--sleep"); // something from Flux to make it run for a > while > argList.add("36000000"); > > String[] ecoArgs = argList.toArray(new String[argList.size()]); > > Eco.main(ecoArgs); > > } > } > > > -- > Sent from A Mobile Device > >
