Hello, Jose thank you for your useful response. I guess that is a better approach and works.
Thanks, Nikos 2015-02-18 1:11 GMT-05:00 Andrew Xor <[email protected]>: > I concur to this; as a rule of the thumb I use compile when I develop > locally (in LocalCluster) and provided if I deploy the topology to an > actual cluster. > > Regards. > > On Wed, Feb 18, 2015 at 7:23 AM, José Barrueta <[email protected]> wrote: > >> Hey Nikon, I had the same issue days ago, the problem in my case was that >> I wasn't declaring the storm-core dependency as provided in the pom file, >> therefore the fat jar included the dependency, which is provided by nimbus >> (and workers) when deploying in the cluster. >> >> ... >> <dependency> >> <groupId>org.apache.storm</groupId> >> <artifactId>storm-core</artifactId> >> <version>${storm.version}</version> >> <scope>provided</scope> >> </dependency> >> ... >> >> Best, >> >> Jose Luis >> >> On Tue, Feb 17, 2015 at 5:29 PM, Nick R. Katsipoulakis < >> [email protected]> wrote: >> >>> Hello, >>> >>> I am currently trying to automate some experiments on Storm and to do >>> so, I have created a Java process which in turn launches three Java >>> sub-processes using the ProcessBuilder framework. The three processes are >>> mainly executions of jar files (2 of them simple java applications, 1 of >>> them Storm topology submitted to a Storm cluster). Therefore, the 2 simple >>> java applications are executed as terminal commands: >>> >>> $>java -jar executable-jar <arg-1> <arg-2> ... <arg-N> >>> >>> The storm topology is bundled into a jar using Eclipse maven package >>> assembly:single goal and the following plugin configuration is added to the >>> pom.xml of the project: >>> >>> <build> >>> >>> <plugins> >>> >>> <plugin> >>> >>> <groupId>org.apache.maven.plugins</groupId> >>> >>> <artifactId>maven-assembly-plugin</artifactId> >>> >>> <version>2.5.3</version> >>> >>> <configuration> >>> >>> <descriptorRefs> >>> >>> <descriptorRef>jar-with-dependencies</descriptorRef> >>> >>> </descriptorRefs> >>> >>> <archive> >>> >>> <manifest> >>> >>> <mainClass>gr.katsip.storm.topology.ExperimentalTopology</mainClass> >>> >>> </manifest> >>> >>> </archive> >>> >>> </configuration> >>> >>> <executions> >>> >>> <execution> >>> >>> <phase>package</phase> >>> >>> <goals> >>> >>> <goal>single</goal> >>> >>> </goals> >>> >>> </execution> >>> >>> </executions> >>> >>> </plugin> >>> >>> </plugins> >>> >>> </build> >>> >>> >>> The problem is that when I create the Java sub-process that submits the >>> following command: >>> >>> >>> $>storm jar bundled-topology.jar <arg-1> <arg-2> ... <arg-M> >>> >>> >>> at some point I get the following error: >>> >>> >>> Exception in thread "main" java.lang.RuntimeException: Found multiple >>> defaults.yaml resources. You're probably bundling the Storm jars with your >>> topology jar. >>> [jar:file:/Users/nick/Documents/stream-gen/Experiment/synefo-topology.jar!/defaults.yaml, >>> jar:file:/Users/nick/apache-storm/lib/storm-core-0.9.2-incubating.jar!/defaults.yaml] >>> >>> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140) >>> >>> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167) >>> >>> at backtype.storm.utils.Utils.readStormConfig(Utils.java:191) >>> >>> at backtype.storm.config$read_storm_config.invoke(config.clj:121) >>> >>> at backtype.storm.command.config_value$_main.invoke(config_value.clj:22) >>> >>> at clojure.lang.AFn.applyToHelper(AFn.java:161) >>> >>> at clojure.lang.AFn.applyTo(AFn.java:151) >>> >>> at backtype.storm.command.config_value.main(Unknown Source) >>> >>> Therefore, I understand that during packaging, maven includes a >>> default.yaml file in my package, that conflicts with the yam file of the >>> cluster. >>> >>> My question is how can I remove the default.yaml from my >>> bundled-topology or how can I overcome this issue. The simple solution >>> would be to start my topology by itself and then run the other two >>> processes, but I want my experiments to be fully automated. >>> >>> Thank you and I apologize in advance for the long post. >>> >>> Cheers, >>> Nikos >>> >>> -- >>> Nikolaos Romanos Katsipoulakis, >>> University of Pittsburgh, PhD candidate >>> >> >> > -- Nikolaos Romanos Katsipoulakis, University of Pittsburgh, PhD candidate
