Re: Zero Worker Bug in Topology on 1.0.2

[Jacob Johansen]

Are your supervisors still running?


On November 22, 2016 at 2:46:08 PM, P. Taylor Goetz (ptgo...@gmail.com) wrote:

What does the Storm UI show? Are you out of slots?

-Taylor


On Nov 21, 2016, at 6:28 PM, Joaquin Menchaca  wrote:

I am not sure what is causing this, but it seems that after 70 days of running 
the cluster, it will not be able to run a topology with any workers or tasks.

I tried a sample topology, which used to work, but now it is broken?

storm jar 
/usr/lib/apache/storm/1.0.2/examples/storm-starter/storm-starter-topologies-1.0.2.jar
 org.apache.storm.starter.ExclamationTopology ExclamationTopology


Topology_name        Status     Num_tasks  Num_workers  Uptime_secs
---
ExclamationTopology   ACTIVE     0          0            9        

Any suggestions?

--

是故勝兵先勝而後求戰，敗兵先戰而後求勝。

Re: Zero Worker Bug in Topology on 1.0.2

[P. Taylor Goetz]

What does the Storm UI show? Are you out of slots?

-Taylor


> On Nov 21, 2016, at 6:28 PM, Joaquin Menchaca  wrote:
> 
> I am not sure what is causing this, but it seems that after 70 days of 
> running the cluster, it will not be able to run a topology with any workers 
> or tasks.
> 
> I tried a sample topology, which used to work, but now it is broken?
> 
> storm jar 
> /usr/lib/apache/storm/1.0.2/examples/storm-starter/storm-starter-topologies-1.0.2.jar
>  org.apache.storm.starter.ExclamationTopology ExclamationTopology
> 
> 
> Topology_nameStatus Num_tasks  Num_workers  Uptime_secs
> ---
> ExclamationTopology   ACTIVE 0  09
> 
> Any suggestions?
> 
> -- 
> 
> 是故勝兵先勝而後求戰，敗兵先戰而後求勝。

CEP

[Mitz Leiner]

Hi!
I would like to apply CEP to built an autonomous model factory. I have
already found a lot on the documentation but still have some questions. It
would be great if some of you could help me with those:

- What components are distributed by the open source software? Is there a
way to perform complex event processing with apache storm, other than
adding an external cep engine like esper or drools to it?
- how well is Apache Storm scalable (horizontally)? how can you measure
scalability?
- how stable is the event throughput by adding more rules?
- What knowledge is needed to use Apache storm? How easy is it to maintain
the system once it is set up?
- How powerful is the EPL?
- What data formats are supported?
- How many times has Apache Storm been downloaded? Or do you have any
numbers available of how often it is used not only for ESP but also CEP?
- Are there any analytical  tools available? Is there a reporting system
which, e.g., exports analytic results as .pdf?
- How reliable are the results?

Many thanks in advance,

M

RE: Containerising Storm

[Marc Roos]

 
What about having a look at Heron? Afaik it was developed specifically 
to be run with an external scheduling component. 




- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. 
F1 Outsourcing Development Sp. z o.o.
Poland 

t:  +48 (0)124466845
f:  +48 (0)124466843
e:  m...@f1-outsourcing.eu


-Original Message-
From: Zachary Smith [mailto:tud20...@temple.edu] 
Sent: dinsdag 22 november 2016 19:28
To: user@storm.apache.org
Subject: Re: Containerising Storm

storm-mesos also provides a way to use storm on top of mesos/marathon 

https://github.com/mesos/storm


On Tue, Nov 22, 2016 at 12:52 PM, Ambud Sharma  
wrote:


https://github.com/Symantec/hendrix/blob/current/install/scripts/lo
cal-build-install-docker.sh 

 

Here's a docker compose way of doing it. Zookeeper in this case is 
not scalable.

I had to rebuild some of the images for Storm to fix issues with 
environment variables and DNS lookups. Those images should work with 
kubernetes as well.


On Nov 22, 2016 9:48 AM, "Cuneo, Nicholas"  
wrote:


We tried doing the same but ran into deployment issues with 
storm workers being deployed on separate docker instances. I wasn't 
directly involved so I can't give much details, I just know that storm 
is no longer containerized on our end.

---
Sent from Boxer | http://getboxer.com  


On November 22, 2016 at 9:08:53 AM PST, Al Hennessey 
 wrote:


Hi
I am looking to containerise my storm setup so I can 
run 
it across services like kubernetes or mesos. I can't find an official 
docker release and wondering what you guys recommend or whether it's 
easier just to package one myself.

Thanks for the help





This e-mail contains privileged and confidential information 
intended for the use of the addressees named above. If you are not the 
intended recipient of this e-mail, you are hereby notified that you must 
not disseminate, copy or take any action in respect of any information 
contained in it. If you have received this e-mail in error, please 
notify the sender immediately by e-mail and immediately destroy this 
e-mail and its attachments.

Re: Containerising Storm

[Zachary Smith]

storm-mesos also provides a way to use storm on top of mesos/marathon

https://github.com/mesos/storm

On Tue, Nov 22, 2016 at 12:52 PM, Ambud Sharma 
wrote:

> https://github.com/Symantec/hendrix/blob/current/install/
> scripts/local-build-install-docker.sh
>
> Here's a docker compose way of doing it. Zookeeper in this case is not
> scalable.
>
> I had to rebuild some of the images for Storm to fix issues with
> environment variables and DNS lookups. Those images should work with
> kubernetes as well.
>
> On Nov 22, 2016 9:48 AM, "Cuneo, Nicholas"  wrote:
>
>> We tried doing the same but ran into deployment issues with storm workers
>> being deployed on separate docker instances. I wasn't directly involved so
>> I can't give much details, I just know that storm is no longer
>> containerized on our end.
>>
>> ---
>> Sent from Boxer | http://getboxer.com 
>>
>> On November 22, 2016 at 9:08:53 AM PST, Al Hennessey <
>> alhennesse...@gmail.com> wrote:
>>
>> Hi
>> I am looking to containerise my storm setup so I can run it across
>> services like kubernetes or mesos. I can't find an official docker release
>> and wondering what you guys recommend or whether it's easier just to
>> package one myself.
>>
>> Thanks for the help
>>
>>
>> --
>>
>> This e-mail contains privileged and confidential information intended for
>> the use of the addressees named above. If you are not the intended
>> recipient of this e-mail, you are hereby notified that you must not
>> disseminate, copy or take any action in respect of any information
>> contained in it. If you have received this e-mail in error, please notify
>> the sender immediately by e-mail and immediately destroy this e-mail and
>> its attachments.
>>
>

Re: Containerising Storm

[Ambud Sharma]

https://github.com/Symantec/hendrix/blob/current/install/scripts/local-build-install-docker.sh

Here's a docker compose way of doing it. Zookeeper in this case is not
scalable.

I had to rebuild some of the images for Storm to fix issues with
environment variables and DNS lookups. Those images should work with
kubernetes as well.

On Nov 22, 2016 9:48 AM, "Cuneo, Nicholas"  wrote:

> We tried doing the same but ran into deployment issues with storm workers
> being deployed on separate docker instances. I wasn't directly involved so
> I can't give much details, I just know that storm is no longer
> containerized on our end.
>
> ---
> Sent from Boxer | http://getboxer.com 
>
> On November 22, 2016 at 9:08:53 AM PST, Al Hennessey <
> alhennesse...@gmail.com> wrote:
>
> Hi
> I am looking to containerise my storm setup so I can run it across
> services like kubernetes or mesos. I can't find an official docker release
> and wondering what you guys recommend or whether it's easier just to
> package one myself.
>
> Thanks for the help
>
>
> --
>
> This e-mail contains privileged and confidential information intended for
> the use of the addressees named above. If you are not the intended
> recipient of this e-mail, you are hereby notified that you must not
> disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please notify
> the sender immediately by e-mail and immediately destroy this e-mail and
> its attachments.
>

Re: Containerising Storm

[Cuneo, Nicholas]

We tried doing the same but ran into deployment issues with storm workers being 
deployed on separate docker instances. I wasn't directly involved so I can't 
give much details, I just know that storm is no longer containerized on our end.

---
Sent from Boxer | http://getboxer.com

On November 22, 2016 at 9:08:53 AM PST, Al Hennessey  
wrote:
Hi
I am looking to containerise my storm setup so I can run it across services 
like kubernetes or mesos. I can't find an official docker release and wondering 
what you guys recommend or whether it's easier just to package one myself.

Thanks for the help



This e-mail contains privileged and confidential information intended for the 
use of the addressees named above. If you are not the intended recipient of 
this e-mail, you are hereby notified that you must not disseminate, copy or 
take any action in respect of any information contained in it. If you have 
received this e-mail in error, please notify the sender immediately by e-mail 
and immediately destroy this e-mail and its attachments.

Re: Containerising Storm

[Paritosh Anand]

Hi,

There is an official image for Apache Storm available on Docker hub. You
can start by trying this image.

If you may, you can also try this image
https://hub.docker.com/r/kakapari/docker-storm/ this simply installs

1. Zookeeper v3.4.8
2. Storm nimbus v0.10.1
3. Storm supervisor with 2 workers.

It is easier if you go through the Dockerfile for these repo and you might
find easy to write your own as well.

P

On 22 November 2016 at 22:38, Al Hennessey  wrote:

> Hi
> I am looking to containerise my storm setup so I can run it across
> services like kubernetes or mesos. I can't find an official docker release
> and wondering what you guys recommend or whether it's easier just to
> package one myself.
>
> Thanks for the help
>

Re: Is LoadAwareShuffleGrouping The Default?

[Kevin Peek]

My apologies, the code I referenced above is old. I believe the correct
references are these:

https://github.com/apache/storm/blob/cd5c9e8f904205a6ca6eee9222ca954ca8b37ec3/storm-core/src/jvm/org/apache/storm/daemon/GrouperFactory.java#L63

https://github.com/apache/storm/blob/master/conf/defaults.yaml#L261

Based on these, I believe the LoadAwareShuffleGrouping is the default. Can
anyone more familiar with the code confirm this?

On Tue, Nov 22, 2016 at 11:43 AM, Kevin Peek  wrote:

> Hello. I have a question about the load aware shuffle grouping.
>
> If I create a grouping like below, will I end up with a
> LoadAwareShuffleGrouping by default?
>
> topologyBuilder.setBolt("boltId", new SomeBolt()).shuffleGrouping("
> otherBoltId");
>
> I would expect this to result in a traditional ShuffleGrouping, but I
> think it does not based on the following:
>
> (1) the config option TOPOLOGY-DISABLE-LOADAWARE-MESSAGING defaults to
> false.
> (2) This block of code seems to create a LoadAwareShuffleGrouping
> automatically if load messaging is enabled (the default).
> https://github.com/revans2/incubator-storm/blob/
> 024ff263800e4032d7b2395fbbf072e0183ebb5b/storm-core/src/clj/
> backtype/storm/daemon/executor.clj#L66
>
>
> Is this correct? Is load aware the default behavior in Storm 1.0.x?
>

Is LoadAwareShuffleGrouping The Default?

[Kevin Peek]

Hello. I have a question about the load aware shuffle grouping.

If I create a grouping like below, will I end up with a
LoadAwareShuffleGrouping by default?

topologyBuilder.setBolt("boltId", new
SomeBolt()).shuffleGrouping("otherBoltId");

I would expect this to result in a traditional ShuffleGrouping, but I think
it does not based on the following:

(1) the config option TOPOLOGY-DISABLE-LOADAWARE-MESSAGING defaults to
false.
(2) This block of code seems to create a LoadAwareShuffleGrouping
automatically if load messaging is enabled (the default).
https://github.com/revans2/incubator-storm/blob/024ff263800e4032d7b2395fbbf072e0183ebb5b/storm-core/src/clj/backtype/storm/daemon/executor.clj#L66


Is this correct? Is load aware the default behavior in Storm 1.0.x?

Support of topic wildcards in Kafka spout

[Wijekoon, Manusha]

We have a topology that has multiple kafka spout tasks. Each spout task is 
supposed to read a subset of messages from a set of Kafka topics. Topics have 
to be subscribed using a wild card such as AAA.BBB.*. The expected behaviour 
would be that all spout tasks collectively will consume all messages in all of 
the topics that match the wild card. Each message is only routed to a single 
spout task (Ignore failure scenarios). Is this currently supported?