I concur to this; as a rule of the thumb I use compile when I develop locally (in LocalCluster) and provided if I deploy the topology to an actual cluster.
Regards. On Wed, Feb 18, 2015 at 7:23 AM, José Barrueta <[email protected]> wrote: > Hey Nikon, I had the same issue days ago, the problem in my case was that > I wasn't declaring the storm-core dependency as provided in the pom file, > therefore the fat jar included the dependency, which is provided by nimbus > (and workers) when deploying in the cluster. > > ... > <dependency> > <groupId>org.apache.storm</groupId> > <artifactId>storm-core</artifactId> > <version>${storm.version}</version> > <scope>provided</scope> > </dependency> > ... > > Best, > > Jose Luis > > On Tue, Feb 17, 2015 at 5:29 PM, Nick R. Katsipoulakis < > [email protected]> wrote: > >> Hello, >> >> I am currently trying to automate some experiments on Storm and to do so, >> I have created a Java process which in turn launches three Java >> sub-processes using the ProcessBuilder framework. The three processes are >> mainly executions of jar files (2 of them simple java applications, 1 of >> them Storm topology submitted to a Storm cluster). Therefore, the 2 simple >> java applications are executed as terminal commands: >> >> $>java -jar executable-jar <arg-1> <arg-2> ... <arg-N> >> >> The storm topology is bundled into a jar using Eclipse maven package >> assembly:single goal and the following plugin configuration is added to the >> pom.xml of the project: >> >> <build> >> >> <plugins> >> >> <plugin> >> >> <groupId>org.apache.maven.plugins</groupId> >> >> <artifactId>maven-assembly-plugin</artifactId> >> >> <version>2.5.3</version> >> >> <configuration> >> >> <descriptorRefs> >> >> <descriptorRef>jar-with-dependencies</descriptorRef> >> >> </descriptorRefs> >> >> <archive> >> >> <manifest> >> >> <mainClass>gr.katsip.storm.topology.ExperimentalTopology</mainClass> >> >> </manifest> >> >> </archive> >> >> </configuration> >> >> <executions> >> >> <execution> >> >> <phase>package</phase> >> >> <goals> >> >> <goal>single</goal> >> >> </goals> >> >> </execution> >> >> </executions> >> >> </plugin> >> >> </plugins> >> >> </build> >> >> >> The problem is that when I create the Java sub-process that submits the >> following command: >> >> >> $>storm jar bundled-topology.jar <arg-1> <arg-2> ... <arg-M> >> >> >> at some point I get the following error: >> >> >> Exception in thread "main" java.lang.RuntimeException: Found multiple >> defaults.yaml resources. You're probably bundling the Storm jars with your >> topology jar. >> [jar:file:/Users/nick/Documents/stream-gen/Experiment/synefo-topology.jar!/defaults.yaml, >> jar:file:/Users/nick/apache-storm/lib/storm-core-0.9.2-incubating.jar!/defaults.yaml] >> >> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140) >> >> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167) >> >> at backtype.storm.utils.Utils.readStormConfig(Utils.java:191) >> >> at backtype.storm.config$read_storm_config.invoke(config.clj:121) >> >> at backtype.storm.command.config_value$_main.invoke(config_value.clj:22) >> >> at clojure.lang.AFn.applyToHelper(AFn.java:161) >> >> at clojure.lang.AFn.applyTo(AFn.java:151) >> >> at backtype.storm.command.config_value.main(Unknown Source) >> >> Therefore, I understand that during packaging, maven includes a >> default.yaml file in my package, that conflicts with the yam file of the >> cluster. >> >> My question is how can I remove the default.yaml from my bundled-topology >> or how can I overcome this issue. The simple solution would be to start my >> topology by itself and then run the other two processes, but I want my >> experiments to be fully automated. >> >> Thank you and I apologize in advance for the long post. >> >> Cheers, >> Nikos >> >> -- >> Nikolaos Romanos Katsipoulakis, >> University of Pittsburgh, PhD candidate >> > >
