Storm use case

2014-09-16 Thread Ayush Vatsyayan
We are trying to build a webservices application that can support *10k TPS*. I'm trying to do some POC's on strom, but I'm a bit concerned if using storm is the right fit here. Here is the scenario: Client will send a webservice request, which we will receive it (using apache CXF) and push it

Decouple topology configuration from code

2014-09-16 Thread dominique.villard
Hi folks, Apache Camel has a number of DSL which allow its topologies (routes wrt. Camel terminology) to be set up and configured easily. I am interested in such approach for Storm. I found java beans usage in: https://github.com/granthenke/storm-spring/ but sounds fairly limited to me. Is

Re: Can I make Nimbus failsafe

2014-09-16 Thread M.Tarkeshwar Rao
Hi all, can you please guide me the Algo or any inputs how i can make nimbus highly available? I have seen the code of JStorm 9.0 where they make nimbus High available using zookeeper. Any one who is done something in it please guide me. regards Tarkeshwar On Wed, Sep 10, 2014 at 2:16 PM,

Re: Trident transactional topology stuck re-emitting batches with Netty, but running fine with ZMQ (was Re: Topology is stuck)

2014-09-16 Thread Danijel Schiavuzzi
Yes, it's been fixed in 'master' for some time now. Danijel On Tuesday, September 16, 2014, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Danijel, Is the issue resolved in any version of the storm? Regards Tarkeshwar On Thu, Jul 17, 2014 at 6:57 PM, Danijel Schiavuzzi

Re: Can I make Nimbus failsafe

2014-09-16 Thread Parth Brahmbhatt
I am working on this right now , can’t promise the dates because I have some other tasks as well. The general approach is already described in the JIRA. You basically need 3 things: Leader election between all nimbus hosts. (zookeeper is what I am going with), checks on nimbus background

Storm 0.9.2-incubating - num workers and num executors switched?

2014-09-16 Thread Tao, Jing
We recently upgraded to Storm 0.9.2-incubating, and found that on the UI, Num workers and Num executors switched. Example: In older version (0.9.0.1): [cid:image001.png@01CFD1C4.B986FB20] In new version (0.9.2-incubating): [cid:image002.png@01CFD1C4.B986FB20] Is this a UI bug? Or did

Re: Storm 0.9.2-incubating - num workers and num executors switched?

2014-09-16 Thread Harsha
Hi Jing, Its the UI bug fixed in the trunk. [1]https://issues.apache.org/jira/browse/STORM-369 -Harsha On Tue, Sep 16, 2014, at 12:45 PM, Tao, Jing wrote: We recently upgraded to Storm 0.9.2-incubating, and found that on the UI, Num workers and Num executors switched. Example: In

Trying to run test Storm App on Windows but getting problems with POM file

2014-09-16 Thread Gezim Musliaj
[INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for

muliple-nodes kafka cluster

2014-09-16 Thread Sa Li
Hi, All I have been using kafka cluster in single server with three brokers, but I am thinking to build a larger kafka cluster, say 4 nodes (server), and 3 brokers in each node, so totally 12 brokers, would that be better than single node cluster? Or single node will be fair enough, since web api

Re: muliple-nodes kafka cluster

2014-09-16 Thread Harsha
Hi Alec, Single node kafka cluster not recommended apart from using it for development. I highly recommend using multinode cluster and create a partitioned topic with replication. This not only makes it optimal to take in more data at faster rates also allows your cluster running if there

Re: muliple-nodes kafka cluster

2014-09-16 Thread Gezim Musliaj
[INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is

Re: Trying to run test Storm App on Windows but getting problems with POM file

2014-09-16 Thread Nick Beenham
I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO]

Re: Trying to run test Storm App on Windows but getting problems with POM file

2014-09-16 Thread Gezim Musliaj
I have been following these instructions: If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as Could not resolve dependencies for

Re: Decouple topology configuration from code

2014-09-16 Thread Corey Nolet
Awhile ago I had written a camel adapter for storm so that spout inputs could come from camel. Not sure how useful it would be for you but its located here: https://github.com/calrissian/storm-recipes/blob/master/camel/src/main/java/org/calrissian/recipes/camel/spout/CamelConsumerSpout.java Hi

Re: Decouple topology configuration from code

2014-09-16 Thread Corey Nolet
Also, Trident is a DSL for rapidly producing useful analytics in Storm and I've been working on a DSL that makes streams processing for complex event processing possible. That one is located here: https://github.com/calrissian/flowmix On Sep 16, 2014 4:29 AM, dominique.vill...@orange.com wrote:

Re: Trying to run test Storm App on Windows but getting problems with POM file

2014-09-16 Thread Harsha
Hi , Did you changed storm/pom.xml groupIdorg.apache.storm/groupId artifactIdstorm/artifactId version0.9.3-incubating-SNAPSHOT/version and also are your running mvn install from top-level dir not from storm-starter. -Harsha On Tue, Sep 16, 2014, at 03:12 PM, Gezim

Re: Trying to run test Storm App on Windows but getting problems with POM file

2014-09-16 Thread Gezim Musliaj
Yes, Im running it from top level dir (incubator-storm) with the following command: mvn clean install -DskipTests=true and still cant get it to work properly, I have tried to build it from the storm-starter and still the same, keeps telling me there is an error with dependencies. On Wed, Sep 17,

Is anyone using Scala/ ScalaStorm to write Storm Bolts?

2014-09-16 Thread Ash G
have you written Storm Bolts in Scala and used in production? Did you use https://github.com/velvia/ScalaStorm How was the experience?

Re: Trying to run test Storm App on Windows but getting problems with POM file

2014-09-16 Thread Gezim Musliaj
Finally I solved the problem, I had tried building storm-starter before the main storm project. I just removed the previous folder with all of its files, extracted a new one from the incubator-storm.rar and did the build succesfully. But when I make the eclipse project files via the command mvn

Re: Trident transactional topology stuck re-emitting batches with Netty, but running fine with ZMQ (was Re: Topology is stuck)

2014-09-16 Thread M.Tarkeshwar Rao
Thanks for reply.can I pulll this fix or can I download it? On 17 Sep 2014 09:58, tarkeshwa...@gmail.com wrote: In which version it is available. On 16 Sep 2014 19:01, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: Yes, it's been fixed in 'master' for some time now. Danijel On Tuesday,

Re: Trident transactional topology stuck re-emitting batches with Netty, but running fine with ZMQ (was Re: Topology is stuck)

2014-09-16 Thread M.Tarkeshwar Rao
In which version it is available. On 16 Sep 2014 19:01, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: Yes, it's been fixed in 'master' for some time now. Danijel On Tuesday, September 16, 2014, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Danijel, Is the issue resolved in any