Hi,
I am grad student, and I am working on Storm as a part of my course project.
I have added some code in bolt-emit function, which has increased the
latency.
Is this expected?
(fast-list-iter [^TupleImpl a anchors]
(let
[ root-ids (-> a .getMessageId .getAnchorsToIds .keySet); D1-messageId
ack-val (.getAckVal a)]
(when (pos? (count root-ids))
(.updateAckVal a 1)
(log-message
"CurrTupleCount: " (.getCurrTupleCount a))
(if (= 1 (.get task_counter))
(.incrementCurrTupleCount a))
(log-message "CurrTupleCount: " (.getCurrTupleCount a))
(fast-map-iter [[root-id weight] (.. a getMessageId getAnchorsToIds)]
(log-message "HAHA1: root id" root-id " CurrTupleCount: "
(.getCurrTupleCount a))
(let [curr-task-count (.get task_counter)
bIsLastTask? (= curr-task-count (count out-tasks))
tokens_tobe_sent (.calTokens a weight tuple_count (count
out-tasks) bIsLastTask?)]
(put-add! anchors-to-ids root-id tokens_tobe_sent)
(.updateLeftWeightsMap a root-id tokens_tobe_sent)
(log-message "VIBHA-DEBUG: Bolt[2] Component id: " component-id
" weight debug: " root-id " Input weight: " weight " Output weight:"
tokens_tobe_sent "task_counter:" curr-task-count)))
))) ;fast-list-iter ends.
(let [tuple
(TupleImpl. worker-context
values
task-id
stream
(MessageId/makeId anchors-to-ids))]
(transfer-fn t
tuple))))
In the above code, calTokens and updateLeftWeightsMap are java functions.
calTokens do a '/' and '%', while updateLeftWeightsMap updates the
calculated value in the hash.
Can anybody please tell if there is any optimized way that I am not aware
of?
Thanks already!!
Vibha Goyal
On Mon, Mar 14, 2016 at 11:23 AM, Harsha <[email protected]> wrote:
> Do not package storm-core in your topology. Make sure the scope is set to
> provided.
>
>
> On Mon, Mar 14, 2016, at 07:36 AM, vibha goyal wrote:
>
>
> Thank you all!!
>
> How do I exclude this from topology? I have not made any change in any
> file.
>
>
> Just using the default files, and creating the jar files using "mvn clean
> install' in storm-starter.
>
> Regards,
> Vibha Goyal
>
> On Mon, Mar 14, 2016 at 2:58 AM, 马哲超 <[email protected]> wrote:
>
> *stom.yaml* is uesed when starting the storm cluster, not your topology.
> Do not package this file into your topology.
>
> 2016-03-14 15:15 GMT+08:00 vibha goyal <[email protected]>:
>
> Thanks, but as I mentioned in the original e-mail, I cannot keep
> storm.yaml in /home/vgoyal5/apache-storm-1.0.0-SNAPSHOT/conf/storm.yaml" .
>
> If i keep this, it give me error.
>
> How to make changes to yaml file?
>
> On Mon, Mar 14, 2016 at 1:30 AM, Harsha <[email protected]> wrote:
>
>
> its for Nimbus HA
> http://hortonworks.com/blog/fault-tolerant-nimbus-in-apache-storm/
>
>
> On Sun, Mar 13, 2016, at 11:27 PM, Sai Dilip Reddy Kiralam wrote:
>
>
> Hi Harsha,
> can you explain why nimbus seeds are used - [nimbus.seeds: ["host1",
> "host2", "host3"];]
>
>
>
>
>
> On Mon, Mar 14, 2016 at 9:49 AM, Harsha <[email protected]> wrote:
>
>
> Vibha,
> For multi-node cluster the main config we need are
> nimbus.seeds: ["nimbus_host_name"]
> storm.zookeeper.servers:
> - "zookeeper_host_name"
> - "zookeeper_host_name"
>
> You can look at the example here
> https://github.com/apache/storm/blob/master/conf/storm.yaml.example
>
> make sure you copy the same storm.yaml in all the nodes in the storm
> cluster.
>
>
> -Harsha
>
>
>
> On Sun, Mar 13, 2016, at 05:07 PM, Xiang Wang wrote:
>
> Hi,
>
> I guess you are in the wrong directory. Do "mvn package" under
> "$STORM_HOME/examples/storm-starter/". Then you can find a jar like
> "storm-starter-0.10.0.jar" in the target directory, which should be
> uploaded to storm cluster.
>
>
>
>
> -------------------------------
> Xiang Wang PhD Candidate
> Database Research Group
>
> School of Computer Science and Engineering
>
> The University of New South Wales
>
> Sydney, Australia
>
> On Mon, Mar 14, 2016 at 7:43 AM, vibha goyal <[email protected]> wrote:
>
> Hi,
>
> I am grad student, and I am working on Storm as a part of my course
> project.
>
> I have cloned the source code from git, and followed the instructions.
>
> I am using branch : 0.10.0
>
> After "mvn package", when I copy the apache-storm-<version>.tar.gz in my
> home,
> untar it and try to run nimbus.
>
> I get error:
>
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at
> org.apache.storm.utils.ConfigUtils.readStormConfigImpl(ConfigUtils.java:138)
> at org.apache.storm.utils.ConfigUtils.readStormConfig(ConfigUtils.java:134)
> at org.apache.storm.command.ConfigValue.main(ConfigValue.java:27)
> Caused by: java.lang.RuntimeException: java.io.IOException: Found multiple
> storm.yaml resources. You're probably bundling the Storm jars with your
> topology jar.
> [jar:file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/lib/storm-core-2.0.0-SNAPSHOT.jar!/storm.yaml,
> file:/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml]
> at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:372)
> at org.apache.storm.utils.Utils.readStormConfig(Utils.java:456)
> at org.apache.storm.utils.Utils.<clinit>(Utils.java:173)
> ... 3 more
>
>
> I deleted the "/home/vgoyal5/apache-storm-2.0.0-SNAPSHOT/conf/storm.yaml"
> file and again tried.
>
> This time I was able to connect the nimbus and supervisors and could see
> the connections in UI.
>
> As soon as I tried to submit the topology with the command:
>
>
>
> storm jar examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar
> org.apache.storm.starter.ExclamationTopology exclamation-topology
>
>
> I got error:
>
> org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find
> leader nimbus from seed hosts [sp16-cs525-g05-01.cs.illinois.edu]. Did
> you specify a valid list of nimbus hosts for config nimbus.seeds?
>
> First ,I am not able to resolve this error,
>
> and second, I cannot make changes in storm.yaml as I deleted the
> conf/storm.yaml
>
> I am a beginner in Storm. I have been trying this for a day now. Any help
> would be appreciated!!
>
>
>
> Thanks!
>
>
> Vibha Goyal
>
>
>
>
>
>
>
>
