Hello,
I am currently trying to automate some experiments on Storm and to do so, I
have created a Java process which in turn launches three Java sub-processes
using the ProcessBuilder framework. The three processes are mainly
executions of jar files (2 of them simple java applications, 1 of them
Storm topology submitted to a Storm cluster). Therefore, the 2 simple java
applications are executed as terminal commands:
$>java -jar executable-jar <arg-1> <arg-2> ... <arg-N>
The storm topology is bundled into a jar using Eclipse maven package
assembly:single goal and the following plugin configuration is added to the
pom.xml of the project:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.3</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>gr.katsip.storm.topology.ExperimentalTopology</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
The problem is that when I create the Java sub-process that submits the
following command:
$>storm jar bundled-topology.jar <arg-1> <arg-2> ... <arg-M>
at some point I get the following error:
Exception in thread "main" java.lang.RuntimeException: Found multiple
defaults.yaml resources. You're probably bundling the Storm jars with your
topology jar.
[jar:file:/Users/nick/Documents/stream-gen/Experiment/synefo-topology.jar!/defaults.yaml,
jar:file:/Users/nick/apache-storm/lib/storm-core-0.9.2-incubating.jar!/defaults.yaml]
at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140)
at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:191)
at backtype.storm.config$read_storm_config.invoke(config.clj:121)
at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.AFn.applyTo(AFn.java:151)
at backtype.storm.command.config_value.main(Unknown Source)
Therefore, I understand that during packaging, maven includes a
default.yaml file in my package, that conflicts with the yam file of the
cluster.
My question is how can I remove the default.yaml from my bundled-topology
or how can I overcome this issue. The simple solution would be to start my
topology by itself and then run the other two processes, but I want my
experiments to be fully automated.
Thank you and I apologize in advance for the long post.
Cheers,
Nikos
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
