Hi Scott, I think we would need to make changes to the EcoSubmitter class to allow for use of the Simulator. How would you prefer to specify that topology would be run in the simulator? Would you prefer to specify this in the topology yaml file or would you rather specify it as a command line argument, something like """ heron submit local \ some/path/to/jar.jar \ org.apache.heron.eco.Eco \ --eco path/to/topology.yaml --simulate """ If we pass --simulate as a parameter for the Eco deployment I have a concern that it could be confusing for the other deployment methods since it would be ignored for something like the Java Topology API as we would specify this in the java code, not a deployment parameter.
- Josh On Sat, Jun 5, 2021 at 2:10 PM Josh Fischer <[email protected]> wrote: > Hi Scott, > > > I’ll have time to look in to this tomorrow for you. I’ve also sent this to > the dev@heron list in case someone else would be interested. > > - Josh > > > On Fri, Jun 4, 2021 at 2:30 PM Scott Mayne <[email protected]> > wrote: > >> Hi all, >> >> I asked a question about running ECO in local mode a while back, thanks >> to Josh for suggesting the verbose flag but nothing was added to the >> console that shed any light on why it exited right away. >> >> In the meantime, I got things running in IntelliJ >> using org.apache.heron.simulator.Simulator, after creating the topology >> using org.apache.heron.api.topology.TopologyBuilder. >> >> Now that that's working, we've circled back to the ECO/IDE/YAML question. >> We have a bunch of Storm 1.0.6 ETLs that we run using command line Flux for >> cluster deploys, and in an IDE with a main() method using Flux as well (see >> bottom of message for my unsuccessful attempt to mimic the Flux way with >> Eco). >> >> Obviously Storm 1.0.6 is getting a little long in the tooth and we're >> exploring moving to Heron (and liking what we see so far). >> >> A nice thing about the Storm setup was we use one YAML file to define the >> topologies in both scenarios. So far in Heron, it would be ECO/YAML for >> cluster (not that we've gotten that far yet) and TopologyBuilder in the >> IDE for development. >> >> Would be nice to not have this difference and use YAML for both (avoid >> mismatches between the two, etc.), so my question is: Has anyone set up >> Heron in IntelliJ (or other IDE) so it can read a (strictly Heron) topology >> from a YAML file and run, hit breakpoints, etc. (with ECO or otherwise)? If >> so, how? >> >> As background, these Storm ETLs were written before I joined the team, >> and I'm relatively new to streaming frameworks, so I might be missing >> something obvious here. >> >> For example, I just removed the Storm artifacts from the POM, added >> Heron, and modified the ETL code as needed. Haven't installed Heron >> binaries or anything like that on my laptop, not sure if that's a >> prerequisite. >> >> Thanks, >> Scott >> >> public class KafkaTopology { >> >> public static void main(String[] args) throws Exception { >> >> List<String> argList = new ArrayList<String>(Arrays.asList(args)); >> String file = >> KafkaTopology.class.getClassLoader().getResource("topology.yaml").getFile(); >> argList.add("local"); >> argList.add("--eco-config-file"); >> argList.add(file); >> >> file = >> KafkaTopology.class.getClassLoader().getResource("dev.properties").getFile(); >> argList.add("--props"); >> argList.add(file); >> >> argList.add("--sleep"); // something from Flux to make it run for a >> while >> argList.add("36000000"); >> >> String[] ecoArgs = argList.toArray(new String[argList.size()]); >> >> Eco.main(ecoArgs); >> >> } >> } >> >> >> -- > Sent from A Mobile Device >
