Hi Josh, My first reaction is for it to not be specified in the topology YAML (seems easier to handle if nothing changes in that file, however deployed).
But it's certainly not a dealbreaker and will defer to your opinion as to what's best from an overall Heron point of view. This new Simulator functionality in EcoSubmitter, it will allow an environment-specific HeronConfig file to be passed in? And the Flux syntax didn't need a JAR path with the "local" invocation, just to reiterate the goal here is to have Eco use the IDE workspace files (Java files for spouts, bolts, etc.) to allow breakpoints, etc. Thanks, Scott ________________________________ From: Josh Fischer <[email protected]> Sent: June 6, 2021 10:10 PM To: [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]> Subject: Re: ECO + YAML in IDE Hi Scott, I think we would need to make changes to the EcoSubmitter class to allow for use of the Simulator. How would you prefer to specify that topology would be run in the simulator? Would you prefer to specify this in the topology yaml file or would you rather specify it as a command line argument, something like """ heron submit local \ some/path/to/jar.jar \ org.apache.heron.eco.Eco \ --eco path/to/topology.yaml --simulate """ If we pass --simulate as a parameter for the Eco deployment I have a concern that it could be confusing for the other deployment methods since it would be ignored for something like the Java Topology API as we would specify this in the java code, not a deployment parameter. - Josh On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <[email protected]<mailto:[email protected]>> wrote: Hi Scott, I’ll have time to look in to this tomorrow for you. I’ve also sent this to the dev@heron list in case someone else would be interested. - Josh On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <[email protected]<mailto:[email protected]>> wrote: Hi all, I asked a question about running ECO in local mode a while back, thanks to Josh for suggesting the verbose flag but nothing was added to the console that shed any light on why it exited right away. In the meantime, I got things running in IntelliJ using org.apache.heron.simulator.Simulator, after creating the topology using org.apache.heron.api.topology.TopologyBuilder. Now that that's working, we've circled back to the ECO/IDE/YAML question. We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for cluster deploys, and in an IDE with a main() method using Flux as well (see bottom of message for my unsuccessful attempt to mimic the Flux way with Eco). Obviously Storm 1.0.6 is getting a little long in the tooth and we're exploring moving to Heron (and liking what we see so far). A nice thing about the Storm setup was we use one YAML file to define the topologies in both scenarios. So far in Heron, it would be ECO/YAML for cluster (not that we've gotten that far yet) and TopologyBuilder in the IDE for development. Would be nice to not have this difference and use YAML for both (avoid mismatches between the two, etc.), so my question is: Has anyone set up Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If so, how? As background, these Storm ETLs were written before I joined the team, and I'm relatively new to streaming frameworks, so I might be missing something obvious here. For example, I just removed the Storm artifacts from the POM, added Heron, and modified the ETL code as needed. Haven't installed Heron binaries or anything like that on my laptop, not sure if that's a prerequisite. Thanks, Scott public class KafkaTopology { public static void main(String[] args) throws Exception { List<String> argList = new ArrayList<String>(Arrays.asList(args)); String file = KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile(); argList.add("local"); argList.add("--eco-config-file"); argList.add(file); file = KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile(); argList.add("--props"); argList.add(file); argList.add("--sleep"); // something from Flux to make it run for a while argList.add("36000000"); String[] ecoArgs = argList.toArray(new String[argList.size()]); Eco.main(ecoArgs); } } -- Sent from A Mobile Device
