And a sample of what I tried, for which I'm still getting the error of:
Caused by: java.lang.RuntimeException: java.io.IOException: Found multiple
defaults.yaml resources. You're probably bundling the Storm jars with your
topology jar.
apply plugin: 'java'
apply plugin: 'eclipse'
defaultTasks 'fatJar'
//create a single Jar with all dependencies
task fatJar(type: Jar) {
version = '0.1.0'
baseName = 'myprogram'
manifest {
attributes 'Implementation-Title': baseName,
'Implementation-Version': version,
'Main-Class' : 'com.slh.myprogram.myprogram'
}
from { (configurations.compile - configurations.provided ).collect {
it.isDirectory() ? it : zipTree(it) } }
with jar
}
sourceCompatibility = '1.8'
[compileJava, compileTestJava]*.options*.encoding = 'UTF-8'
if (!hasProperty('mainClass')) {
ext.mainClass = 'com.slh.myprogram.myprogram'
}
repositories {
mavenCentral()
maven{
url 'http://clojars.org/repo'
}
}
configurations {
provided
compile.extendsFrom provided
}
dependencies {
//---apache storm
compile 'org.apache.storm:storm-core:1.0.1', {
ext {
fatJarExclude = true
}
}
testCompile group: 'junit', name: 'junit', version: '4.10'
}
On Mon, May 23, 2016 at 4:05 PM, Navin Ipe <[email protected]>
wrote:
> Anyone with Gradle experience, kindly help. I've tried for the past 40
> minutes with help from posts like this
> <http://stackoverflow.com/questions/17421726/making-the-storm-jars-compile-time-only-in-a-gradle-project>,
> this
> <http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#build-a-correct-standalone--fat-jar-file-of-my-storm-code>,
> this
> <http://kennethjorgensen.com/blog/2014/fat-jars-with-excluded-dependencies-in-gradle>
> and this
> <https://discuss.gradle.org/t/removing-dependencies-from-a-jar-file-during-jar-task/5521>,
> but it isn't working.
> This is all my original gradle script contains:
>
> apply plugin: 'java'
> apply plugin: 'eclipse'
>
> defaultTasks 'jar'
>
> jar {
> from {
> (configurations.runtime).collect {
> it.isDirectory() ? it : zipTree(it)
> }
> }
> manifest {
> attributes 'Main-Class': 'com.slh.myprogram.MyProgram'
> }
> }
>
> sourceCompatibility = '1.8'
> [compileJava, compileTestJava]*.options*.encoding = 'UTF-8'
>
> if (!hasProperty('mainClass')) {
> ext.mainClass = 'com.slh.myprogram.MyProgram'
> }
>
> repositories {
> mavenCentral()
> }
>
> dependencies {
> compile 'org.apache.storm:storm-core:1.0.1'
> testCompile group: 'junit', name: 'junit', version: '4.10'
> }
>
>
> On Mon, May 23, 2016 at 3:18 PM, Matthias J. Sax <[email protected]> wrote:
>
>> You are doing it basically correct. But you use the wrong jar file...
>>
>> The jar you submit must contain your spouts and bolt classes, but not
>> the classes of Storm itself.
>>
>> I am not too familiar with Gradle, but you need to fix your build script
>> to assemble your topology jar file correctly.
>>
>> It would help, if you post your gradle file here, for other to help you
>> out with this.
>>
>> -Matthias
>>
>> On 05/23/2016 11:45 AM, Navin Ipe wrote:
>> > Bingo. Supervisor.log showed:
>> > /Worker Process 17d197c0-80c8-408d-ac70-11970a93d2b9 exited with code: 1
>> > Worker Process 204c20f5-a6d2-4739-b010-c90e651cb2cf exited with code: 1
>> > 204c20f5-a6d2-4739-b010-c90e651cb2cf still hasn't started
>> > /
>> > And the worker logs were located deep inside here
>> >
>> /Users/nav/programming/apache-storm-1.0.1/*logs/workers-artifacts*/MyTopologyName-6-1463986744/6701/worker.log.err
>> >
>> > *The error:*
>> > /Caused by: java.io.IOException: Found multiple defaults.yaml resources.
>> > You're probably bundling the Storm jars with your topology jar.
>> >
>> [jar:file:/Users/nav/programming/apache-storm-1.0.1/storm-local/supervisor/stormdist/MyTopologyName-6-1463986744/stormjar.jar!/defaults.yaml,
>> >
>> jar:file:/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar!/defaults.yaml]
>> > at
>> > org.apache.storm.utils.Utils.getConfigFileInputStream(Utils.java:333)
>> > at
>> > org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:287)
>> > ... 8 more
>> > Exception in thread "main" java.lang.ExceptionInInitializerError/
>> >
>> > But I don't understand. If I don't include the storm jars in the Gradle
>> > dependency, then I won't be able to compile the program itself.
>> >
>> > *What I'm doing:*
>> > *Using this version of Storm via Gradle in Netbeans.*
>> > dependencies { compile 'org.apache.storm:storm-core:1.0.1'
>> >
>> > *and submitting topology like this:*
>> > System.setProperty("storm.jar",
>> > "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
>> >
>> > StormSubmitter.submitTopology(c.TOPO_NAME, config,
>> > builder.createTopology());
>> >
>> > Is there some other way to do this?
>> >
>> >
>> >
>> > On Mon, May 23, 2016 at 1:36 PM, Matthias J. Sax <[email protected]
>> > <mailto:[email protected]>> wrote:
>> >
>> > How many workers did you configure (ie, storm.yaml parameter
>> > "supervisor.slots.ports" -- the number of available slots is also
>> shown
>> > in the web UI)?
>> >
>> > If there are free workers available, look into logs/worker*.log --
>> if
>> > worker*.log is not available, no worker was started. You might also
>> have
>> > a look into logs/supervisor.log and maybe logs/nimbus.log
>> >
>> >
>> > -Matthias
>> >
>> > On 05/23/2016 08:55 AM, Navin Ipe wrote:
>> > > I installed zookeeper on my local system. Started it. Extracted
>> Storm
>> > > 1.0.1 on my local system and *started it via this script:*
>> > > /#!/bin/bash
>> > > ./storm nimbus &
>> > > ./storm supervisor &
>> > > ./storm ui &/
>> > >
>> > > *storm.yaml has:*
>> > > / storm.zookeeper.servers:
>> > > - "localhost"
>> > > nimbus.seeds: ["localhost"]/
>> > >
>> > > *and zoo.conf has:*
>> > > /tickTime=2000
>> > > initLimit=10
>> > > syncLimit=5
>> > > dataDir=/Users/nav/programming/zookeeper-3.4.8/data
>> > > clientPort=2181/
>> > > *
>> > > *
>> > > *Using this version of Storm via Gradle.*
>> > > /dependencies { compile 'org.apache.storm:storm-core:1.0.1'
>> > >
>> > > /
>> > > *Firewall on my system
>> > > *
>> > > Switched off*
>> > > *
>> > > //
>> > >
>> > > *and submitting topology like this:*
>> > > /System.setProperty("storm.jar",
>> > >
>> "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
>> > >
>> > > StormSubmitter.submitTopology(c.TOPO_NAME, config,
>> > > builder.createTopology());
>> > > /
>> > >
>> > >
>> > > But on running it in my program's build/lib folder with java -jar
>> > > myProgram.jar, the topology gets submitted (and I can see the
>> submitted
>> > > topology on the storm UI via http://localhost:8080),*but after
>> > > submitting, no emits happen. The program seems to freeze.*
>> > > /BUILD SUCCESSFUL
>> > >
>> > > Total time: 12.78 secs
>> > >
>> > > This build could be faster, please consider using the Gradle
>> Daemon:
>> > > https://docs.gradle.org/2.10/userguide/gradle_daemon.html
>> > > 569 [main] INFO c.s.c.s.Queries_ReaderWriter - Waiting few
>> seconds
>> > > until table is dropped...
>> > > 589 [main] INFO c.s.c.s.Queries_ReaderWriter - New table
>> created.
>> > > 1007 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper
>> secret
>> > > payload for MD5-digest: -5943870921993217077:-9035830007806332208
>> > > 1074 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
>> > > 1124 [main] INFO o.a.s.StormSubmitter - Uploading topology jar
>> > >
>> /Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar to
>> > > assigned location:
>> > >
>> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
>> > > 1271 [main] INFO o.a.s.StormSubmitter - Successfully uploaded
>> topology
>> > > jar to assigned location:
>> > >
>> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
>> > > 1271 [main] INFO o.a.s.StormSubmitter - Submitting topology
>> MyTopology
>> > > in distributed mode with conf
>> > >
>> {"storm.zookeeper.topology.auth.scheme":"digest","topology.workers":2,"topology.debug":false,"topology.max.spout.pending":10,"storm.zookeeper.topology.auth.payload":"-5943870921993217077:-9035830007806332208","topology.transfer.buffer.size":32,"topology.executor.send.buffer.size":16384,"topology.executor.receive.buffer.size":16384}
>> > > 1427 [main] INFO o.a.s.StormSubmitter - Finished submitting
>> topology:
>> > > MyTopology
>> > > 1427 [main] INFO c.s.c.CrystalStorm - Topology submitted/
>> > >
>> > > *Two questions:*
>> > > 1. What usually causes this kind of problem? I've seen the
>> answers on
>> > > StackOverflow, but they didn't help.
>> > > 2. Which log files would provide the most useful info in such
>> > situations?
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Navin
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Navin
>>
>>
>
>
> --
> Regards,
> Navin
>
--
Regards,
Navin
