Re: Testing serializers with multiple workers

[Chien Le]

Ohad,


We found that we had to use localOrShuffle grouping in order to see activity in 
the same worker as the spout.


-Chien



From: Ohad Edelstein <oh...@mintigo.com>
Sent: Friday, November 18, 2016 8:38:35 AM
To: user@storm.apache.org
Subject: Re: Testing serializers with multiple workers

Hello,

We just finished setting up storm 1.0.1 with 3 supervisors and one nimbus 
machine.
Total of 4 machines in aws.

We see the following phanomenon:
lets say spout on host2,
host1 - using 100% cpu
host3 - using 100% cpu
host2 - idle (some message are being handled by it, not many)
its not slots problem, we have even amount of bolts.

We also tried to deploy only 2 host, and the same thing happened, the host with 
the spout is idle, the other host at 100% cpu.

We switched from shuffleGrouping to noneGrouping, and its seems to work,
The documentation says that:
None grouping: This grouping specifies that you don't care how the stream is 
grouped. Currently, none groupings are equivalent to shuffle groupings. 
Eventually though, Storm will push down bolts with none groupings to execute in 
the same thread as the bolt or spout they subscribe from (when possible).

We are still trying to understand what is wrong with shuffleGrouping in our 
system,

Any ideas?

Thanks!

From: Aaron Niskodé-Dossett <doss...@gmail.com<mailto:doss...@gmail.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" 
<user@storm.apache.org<mailto:user@storm.apache.org>>
Date: Friday, 18 November 2016 at 17:04
To: "user@storm.apache.org<mailto:user@storm.apache.org>" 
<user@storm.apache.org<mailto:user@storm.apache.org>>
Subject: Re: Testing serializers with multiple workers

Hit send too soon... that really is the option :-)

On Fri, Nov 18, 2016 at 9:03 AM Aaron Niskodé-Dossett 
<doss...@gmail.com<mailto:doss...@gmail.com>> wrote:
topology.testing.always.try.serialize = true

On Fri, Nov 18, 2016 at 8:57 AM Kristopher Kane 
<kkane.l...@gmail.com<mailto:kkane.l...@gmail.com>> wrote:
Does anyone have any techniques for testing serializers that would only surface 
when the serializer is uses in a multi-worker topology?

Kris

Re: Testing serializers with multiple workers

[Ohad Edelstein]

Hello,

We just finished setting up storm 1.0.1 with 3 supervisors and one nimbus 
machine.
Total of 4 machines in aws.

We see the following phanomenon:
lets say spout on host2,
host1 - using 100% cpu
host3 - using 100% cpu
host2 - idle (some message are being handled by it, not many)
its not slots problem, we have even amount of bolts.

We also tried to deploy only 2 host, and the same thing happened, the host with 
the spout is idle, the other host at 100% cpu.

We switched from shuffleGrouping to noneGrouping, and its seems to work,
The documentation says that:
None grouping: This grouping specifies that you don't care how the stream is 
grouped. Currently, none groupings are equivalent to shuffle groupings. 
Eventually though, Storm will push down bolts with none groupings to execute in 
the same thread as the bolt or spout they subscribe from (when possible).

We are still trying to understand what is wrong with shuffleGrouping in our 
system,

Any ideas?

Thanks!

From: Aaron Niskodé-Dossett <doss...@gmail.com<mailto:doss...@gmail.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" 
<user@storm.apache.org<mailto:user@storm.apache.org>>
Date: Friday, 18 November 2016 at 17:04
To: "user@storm.apache.org<mailto:user@storm.apache.org>" 
<user@storm.apache.org<mailto:user@storm.apache.org>>
Subject: Re: Testing serializers with multiple workers

Hit send too soon... that really is the option :-)

On Fri, Nov 18, 2016 at 9:03 AM Aaron Niskodé-Dossett 
<doss...@gmail.com<mailto:doss...@gmail.com>> wrote:
topology.testing.always.try.serialize = true

On Fri, Nov 18, 2016 at 8:57 AM Kristopher Kane 
<kkane.l...@gmail.com<mailto:kkane.l...@gmail.com>> wrote:
Does anyone have any techniques for testing serializers that would only surface 
when the serializer is uses in a multi-worker topology?

Kris

Re: Testing serializers with multiple workers

[Aaron Niskodé-Dossett]

Hit send too soon... that really is the option :-)

On Fri, Nov 18, 2016 at 9:03 AM Aaron Niskodé-Dossett 
wrote:

> topology.testing.always.try.serialize = true
>
> On Fri, Nov 18, 2016 at 8:57 AM Kristopher Kane 
> wrote:
>
> Does anyone have any techniques for testing serializers that would only
> surface when the serializer is uses in a multi-worker topology?
>
> Kris
>
>

Re: Testing serializers with multiple workers

[Aaron Niskodé-Dossett]

topology.testing.always.try.serialize = true

On Fri, Nov 18, 2016 at 8:57 AM Kristopher Kane 
wrote:

> Does anyone have any techniques for testing serializers that would only
> surface when the serializer is uses in a multi-worker topology?
>
> Kris
>