Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

[Rui Abreu]

Hi Thomas!

If you believe it's a bug, please submit it to
https://issues.apache.org/jira/browse/STORM, so hopefully someone will
be able to have a look.

On Mon, 16 Nov 2020 at 15:16, Thomas L. Redman  wrote:

> I think somebody on the Dev team needs to look into this. The topology I
> wrote is on GitHub and should reproduce the problem consistently. This
> feature seems broken.
>
> On Nov 16, 2020, at 4:21 AM, Michael Giroux 
> wrote:
>
> Hello Thomas,
>
> Thank you very much for the response.  Disabling the feature allowed me to
> move from 1.2.3 => 2.2.0.
>
> I understand the intent of the feature however in my case the end result
> was one node getting all the load and eventual netty heap space
> exceptions.  Perhaps I should look at that... or perhaps I'll just leave
> the feature disabled.
>
> Again - thanks for the reply - VERY helpful.
>
>
> On Saturday, November 14, 2020, 01:05:03 PM EST, Thomas L. Redman <
> tomred...@mchsi.com> wrote:
>
>
> I have seen this same thing. I sent a query on this list, and. after some
> time, got a response. The issue is reportedly the result of a new feature.
> I would assume this feature is CLEARLY broken, as I had built a test
> topology that was clearly compute-bound, not IO bound, and there were
> adjustments to change ratio of compute/IO. I have not tested this fix, I am
> too close to release, I just rolled back to version 1.2.3.
>
>  From version 2.1.0 forward, not matter how I changed this compute/net IO
> ratio, tuples were not distributed across nodes. Now, I could only
> reproduce this with anchored (acked) tuples. But if you anchored tuples,
> you could never span more than one node, which defeats the purpose of using
> Storm. Following is that email from Kishor Patil identifying the issue (and
> a way to disable that feature!!!):
>
> *From: *Kishor Patil 
> *Subject: **Re: Significant Bug*
> *Date: *October 29, 2020 at 8:07:18 AM CDT
> *To: *
> *Reply-To: *d...@storm.apache.org
>
> Hello Thomas,
>
> Apologies for delay in responding here. I tested the topology code
> provided in storm-issue repo.
> *only one machine gets peggeg*: Although it appears, his is not a bug.
> This is related to Locality Awareness. Please refer to
> https://github.com/apache/storm/blob/master/docs/LocalityAwareness.md
> It appears spout to bolt ratio is 200, so if there are enough bolt's on
> single node to handle events generated by the spout, it won't send events
> out to another node unless it runs out of capacity on single node. If you
> do not like this and want to distribute events evenly, you can try
> disabling this feature. You can turn off LoadAwareShuffleGrouping by
> setting topology.disable.loadaware.messaging to true.
> -Kishor
>
> On 2020/10/28 15:21:54, "Thomas L. Redman"  wrote:
>
> What’s the word on this? I sent this out some time ago, including a GitHub
> project that clearly demonstrates the brokenness, yet I have not heard a
> word. Is there anybody supporting Storm?
>
> On Sep 30, 2020, at 9:03 AM, Thomas L. Redman  wrote:
>
> I believe I have encountered a significant bug. It seems topologies
> employing anchored tuples do not distribute across multiple nodes,
> regardless of the computation demands of the bolts. It works fine on a
> single node, but when throwing multiple nodes into the mix, only one
> machine gets pegged. When we disable anchoring, it will distribute across
> all nodes just fine, pegging each machine appropriately.
>
> This bug manifests from version 2.1 forward. I first encountered this
> issue with my own production cluster on an app that does significant NLP
> computation across hundreds of millions of documents. This topology is
> fairly complex, so I developed a very simple exemplar that demonstrates the
> issue with only one spout and bolt. I pushed this demonstration up to
> github to provide the developers with a mechanism to easily isolate the
> bug, and maybe provide some workaround. I used gradle to build this simple
> topology and software and package the results. This code is well
> documented, so it should be fairly simple to reproduce the issue. I first
> encountered this issue on 3 32 core nodes, but when I started
> experimenting, I set up a test cluster with 8 cores, and then I increased
> each node to 16 cores, and plenty of memory in every case.
>
> The topology can be accessed from github at
> https://github.com/cowchipkid/storm-issue.git <
> https://github.com/cowchipkid/storm-issue.git>. Please feel free to
> respond to me directory if you have any questions that are beyond the scope
> of this mail list.
>
>
>
>
> Hope this helps. Please let me know how this goes, I will upgrade to 2.2.0
> again for my next release.
>
> On Nov 13, 2020, at 12:53 PM, Michael Giroux 
> wrote:
>
> Hello, all,
>
> I have a topology with 16 workers running across 4 nodes.  This topology
> has a bolt "transform" with executors=1 producing a stream that is comsumed
> by a bolt "ontology" with executors=160.  Everything is configured 

Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

[Paul Jose]

Hi Michael,

Oh alright. Thanks for letting me know what was the issue as well. Cheers.

Best Regards,
Paul

From: Michael Giroux 
Sent: Monday, November 16, 2020 6:03:20 PM
To: user@storm.apache.org 
Subject: Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution 
across cluster

Hello Paul,

Things are sized (1:160) so that the ontology bolts ARE always faster than the 
transform bolts.  Assuming ALL of them are getting workload across the 4 nodes 
in the cluster.

With only 1 node things get overloaded pretty quickly I get netty heap space 
exceptions and still no tuples to the other nodes.   After disabling 
LoadAwareShuffleGrouping by setting topology.disable.loadaware.messaging: true 
all 4 nodes get tuples and all are happy.

FWIW

On Friday, November 13, 2020, 03:07:00 PM EST, Paul Jose 
 wrote:




Hi Michael,

Could it be that the tuples are getting executed much faster on the ontology 
bolt workers than at what rate your transform bolt is executing? Therefore the 
same node would always be ready faster than other workers on other nodes..
If your transform bolt is executing faster than how fast your ontology bolts 
are processing, then I'm not really sure why you're facing this issue..

Best Regards,
Paul

From: Michael Giroux 
Sent: Saturday, November 14, 2020 12:23:43 AM
To: user@storm.apache.org 
Subject: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution 
across cluster

Hello, all,

I have a topology with 16 workers running across 4 nodes.  This topology has a 
bolt "transform" with executors=1 producing a stream that is comsumed by a bolt 
"ontology" with executors=160.  Everything is configured as shufflegrouping.

With Storm 1.2.3 all of the "ontology" bolts get their fair share of tuples.  
When I run Storm 2.2.0 only the "ontology" bolts that are on the same node as 
the single "transform" bolt get tuples.

Same cluster - same baseline code - only difference is binding in the new maven 
artifact.

No errors in the logs.

Any thoughts would be welcome.  Thanks!

---Disclaimer--
Views and opinions expressed in this e-mail belong to their author and do 
not necessarily represent views and opinions of Ugam. Our employees are obliged 
not to make any defamatory statement or infringe any legal right. Therefore, 
Ugam does not accept any responsibility or liability for such statements. The 
content of this email is confidential and intended for the recipient specified 
in message only. It is strictly forbidden to share any part of this message 
with any third party, without a written consent of the sender. If you have 
received this message by mistake, please reply to this message and follow with 
its deletion, so that we can ensure such a mistake does not occur in the 
future. Warning: Sufficient measures have been taken to scan any presence of 
viruses however the recipient should check this email and any attachments for 
the presence of viruses as full security of the email cannot be ensured despite 
our best efforts. Therefore, Ugam accepts no liability for any damage inflicted 
by viewing the content of this email.. 
Please do not print this email unless it is necessary. Every unprinted email 
helps the environment.
---Disclaimer--
 

Views and opinions expressed in this e-mail belong to  their author and do 
not necessarily represent views and opinions  of Ugam. 
Our employees are obliged not to make any defamatory statement or infringe any 
legal right. 
Therefore, Ugam does not accept any responsibility or liability for such 
statements. The content of this email is confidential and intended for the 
recipient specified in message only. It is strictly forbidden to share any part 
of this message with any third party, without a written consent of the sender.
If you have received this message by mistake, please reply to this message and 
follow with its deletion, so that we can ensure such a mistake does not occur 
in the future. 
Warning: Sufficient measures have been taken to scan any presence of viruses 
however the recipient should check this email and any attachments for the 
presence of viruses as full security of the email cannot be ensured despite our 
best efforts.
Therefore, Ugam accepts no liability for any damage inflicted by viewing the 
content of this email.. 

Please do not print this email unless it is necessary. Every unprinted email 
helps the environment. 

Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

[Thomas L. Redman]

I think somebody on the Dev team needs to look into this. The topology I wrote 
is on GitHub and should reproduce the problem consistently. This feature seems 
broken. 

> On Nov 16, 2020, at 4:21 AM, Michael Giroux  
> wrote:
> 
> Hello Thomas,
> 
> Thank you very much for the response.  Disabling the feature allowed me to 
> move from 1.2.3 => 2.2.0.  
> 
> I understand the intent of the feature however in my case the end result was 
> one node getting all the load and eventual netty heap space exceptions.  
> Perhaps I should look at that... or perhaps I'll just leave the feature 
> disabled.
> 
> Again - thanks for the reply - VERY helpful.
> 
> 
> On Saturday, November 14, 2020, 01:05:03 PM EST, Thomas L. Redman 
>  wrote:
> 
> 
> I have seen this same thing. I sent a query on this list, and. after some 
> time, got a response. The issue is reportedly the result of a new feature. I 
> would assume this feature is CLEARLY broken, as I had built a test topology 
> that was clearly compute-bound, not IO bound, and there were adjustments to 
> change ratio of compute/IO. I have not tested this fix, I am too close to 
> release, I just rolled back to version 1.2.3.
> 
>  From version 2.1.0 forward, not matter how I changed this compute/net IO 
> ratio, tuples were not distributed across nodes. Now, I could only reproduce 
> this with anchored (acked) tuples. But if you anchored tuples, you could 
> never span more than one node, which defeats the purpose of using Storm. 
> Following is that email from Kishor Patil identifying the issue (and a way to 
> disable that feature!!!):
> 
>> From: Kishor Patil mailto:kishorvpa...@apache.org>>
>> Subject: Re: Significant Bug
>> Date: October 29, 2020 at 8:07:18 AM CDT
>> To: mailto:d...@storm.apache.org>>
>> Reply-To: d...@storm.apache.org 
>> 
>> Hello Thomas,
>> 
>> Apologies for delay in responding here. I tested the topology code provided 
>> in storm-issue repo. 
>> *only one machine gets peggeg*: Although it appears, his is not a bug. This 
>> is related to Locality Awareness. Please refer to 
>> https://github.com/apache/storm/blob/master/docs/LocalityAwareness.md 
>> 
>> It appears spout to bolt ratio is 200, so if there are enough bolt's on 
>> single node to handle events generated by the spout, it won't send events 
>> out to another node unless it runs out of capacity on single node. If you do 
>> not like this and want to distribute events evenly, you can try disabling 
>> this feature. You can turn off LoadAwareShuffleGrouping by setting 
>> topology.disable.loadaware.messaging to true.
>> -Kishor
>> 
>> On 2020/10/28 15:21:54, "Thomas L. Redman" > > wrote: 
>>> What’s the word on this? I sent this out some time ago, including a GitHub 
>>> project that clearly demonstrates the brokenness, yet I have not heard a 
>>> word. Is there anybody supporting Storm?
>>> 
 On Sep 30, 2020, at 9:03 AM, Thomas L. Redman >>> > wrote:
 
 I believe I have encountered a significant bug. It seems topologies 
 employing anchored tuples do not distribute across multiple nodes, 
 regardless of the computation demands of the bolts. It works fine on a 
 single node, but when throwing multiple nodes into the mix, only one 
 machine gets pegged. When we disable anchoring, it will distribute across 
 all nodes just fine, pegging each machine appropriately.
 
 This bug manifests from version 2.1 forward. I first encountered this 
 issue with my own production cluster on an app that does significant NLP 
 computation across hundreds of millions of documents. This topology is 
 fairly complex, so I developed a very simple exemplar that demonstrates 
 the issue with only one spout and bolt. I pushed this demonstration up to 
 github to provide the developers with a mechanism to easily isolate the 
 bug, and maybe provide some workaround. I used gradle to build this simple 
 topology and software and package the results. This code is well 
 documented, so it should be fairly simple to reproduce the issue. I first 
 encountered this issue on 3 32 core nodes, but when I started 
 experimenting, I set up a test cluster with 8 cores, and then I increased 
 each node to 16 cores, and plenty of memory in every case.
 
 The topology can be accessed from github at 
 https://github.com/cowchipkid/storm-issue.git 
  
 >. Please feel free to 
 respond to me directory if you have any questions that are beyond the 
 scope of this mail list.
>>> 
>>> 
> 
> 
> Hope this helps. Please let me know how this goes, I will upgrade to 2.2.0 
> again for my next release.
> 
>> On Nov 13, 2020, 

Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

[Michael Giroux]

 Hello Paul,
Things are sized (1:160) so that the ontology bolts ARE always faster than the 
transform bolts.  Assuming ALL of them are getting workload across the 4 nodes 
in the cluster.  
With only 1 node things get overloaded pretty quickly I get netty heap space 
exceptions and still no tuples to the other nodes.   After disabling 
LoadAwareShuffleGrouping by setting topology.disable.loadaware.messaging: true 
all 4 nodes get tuples and all are happy.
FWIW
On Friday, November 13, 2020, 03:07:00 PM EST, Paul Jose 
 wrote:  
 
 
 
 Hi Michael,

Could it be that the tuples are getting executed much faster on the ontology 
bolt workers than at what rate your transform bolt is executing? Therefore the 
same node would always be ready faster than other workers on other nodes..
If your transform bolt is executing faster than how fast your ontology bolts 
are processing, then I'm not really sure why you're facing this issue..

Best Regards,
PaulFrom: Michael Giroux 
Sent: Saturday, November 14, 2020 12:23:43 AM
To: user@storm.apache.org 
Subject: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution 
across cluster Hello, all,
I have a topology with 16 workers running across 4 nodes.  This topology has a 
bolt "transform" with executors=1 producing a stream that is comsumed by a bolt 
"ontology" with executors=160.  Everything is configured as shufflegrouping.
With Storm 1.2.3 all of the "ontology" bolts get their fair share of tuples.  
When I run Storm 2.2.0 only the "ontology" bolts that are on the same node as 
the single "transform" bolt get tuples.  
Same cluster - same baseline code - only difference is binding in the new maven 
artifact.
No errors in the logs.  
Any thoughts would be welcome.  Thanks!
---Disclaimer--
 
Views and opinions expressed in this e-mail belong to their author and do 
not necessarily represent views and opinions of Ugam. Our employees are obliged 
not to make any defamatory statement or infringe any legal right. Therefore, 
Ugam does not accept any responsibility or liability for such statements. The 
content of this email is confidential and intended for the recipient specified 
in message only. It is strictly forbidden to share any part of this message 
with any third party, without a written consent of the sender.If you have 
received this message by mistake, please reply to this message and follow with 
its deletion, so that we can ensure such a mistake does not occur in the 
future. Warning: Sufficient measures have been taken to scan any presence of 
viruses however the recipient should check this email and any attachments for 
the presence of viruses as full security of the email cannot be ensured despite 
our best efforts.Therefore, Ugam accepts no liability for any damage inflicted 
by viewing the content of this email.. 
Please do not print this email unless it is necessary. Every unprinted email 
helps the environment. 
 
   

Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

[Michael Giroux]

 Hello Thomas,
Thank you very much for the response.  Disabling the feature allowed me to move 
from 1.2.3 => 2.2.0.  
I understand the intent of the feature however in my case the end result was 
one node getting all the load and eventual netty heap space exceptions.  
Perhaps I should look at that... or perhaps I'll just leave the feature 
disabled.
Again - thanks for the reply - VERY helpful.

On Saturday, November 14, 2020, 01:05:03 PM EST, Thomas L. Redman 
 wrote:  
 
 I have seen this same thing. I sent a query on this list, and. after some 
time, got a response. The issue is reportedly the result of a new feature. I 
would assume this feature is CLEARLY broken, as I had built a test topology 
that was clearly compute-bound, not IO bound, and there were adjustments to 
change ratio of compute/IO. I have not tested this fix, I am too close to 
release, I just rolled back to version 1.2.3.
 From version 2.1.0 forward, not matter how I changed this compute/net IO 
ratio, tuples were not distributed across nodes. Now, I could only reproduce 
this with anchored (acked) tuples. But if you anchored tuples, you could never 
span more than one node, which defeats the purpose of using Storm. Following is 
that email from Kishor Patil identifying the issue (and a way to disable that 
feature!!!):

From: Kishor Patil 
Subject: Re: Significant Bug
Date: October 29, 2020 at 8:07:18 AM CDT
To: 
Reply-To: d...@storm.apache.org

Hello Thomas,

Apologies for delay in responding here. I tested the topology code provided in 
storm-issue repo. 
*only one machine gets peggeg*: Although it appears, his is not a bug. This is 
related to Locality Awareness. Please refer to 
https://github.com/apache/storm/blob/master/docs/LocalityAwareness.md
It appears spout to bolt ratio is 200, so if there are enough bolt's on single 
node to handle events generated by the spout, it won't send events out to 
another node unless it runs out of capacity on single node. If you do not like 
this and want to distribute events evenly, you can try disabling this feature. 
You can turn off LoadAwareShuffleGrouping by setting 
topology.disable.loadaware.messaging to true.
-Kishor

On 2020/10/28 15:21:54, "Thomas L. Redman"  wrote: 

What’s the word on this? I sent this out some time ago, including a GitHub 
project that clearly demonstrates the brokenness, yet I have not heard a word. 
Is there anybody supporting Storm?


On Sep 30, 2020, at 9:03 AM, Thomas L. Redman  wrote:

I believe I have encountered a significant bug. It seems topologies employing 
anchored tuples do not distribute across multiple nodes, regardless of the 
computation demands of the bolts. It works fine on a single node, but when 
throwing multiple nodes into the mix, only one machine gets pegged. When we 
disable anchoring, it will distribute across all nodes just fine, pegging each 
machine appropriately.

This bug manifests from version 2.1 forward. I first encountered this issue 
with my own production cluster on an app that does significant NLP computation 
across hundreds of millions of documents. This topology is fairly complex, so I 
developed a very simple exemplar that demonstrates the issue with only one 
spout and bolt. I pushed this demonstration up to github to provide the 
developers with a mechanism to easily isolate the bug, and maybe provide some 
workaround. I used gradle to build this simple topology and software and 
package the results. This code is well documented, so it should be fairly 
simple to reproduce the issue. I first encountered this issue on 3 32 core 
nodes, but when I started experimenting, I set up a test cluster with 8 cores, 
and then I increased each node to 16 cores, and plenty of memory in every case.

The topology can be accessed from github at 
https://github.com/cowchipkid/storm-issue.git 
. Please feel free to respond to 
me directory if you have any questions that are beyond the scope of this mail 
list.






Hope this helps. Please let me know how this goes, I will upgrade to 2.2.0 
again for my next release.

On Nov 13, 2020, at 12:53 PM, Michael Giroux  wrote:
Hello, all,
I have a topology with 16 workers running across 4 nodes.  This topology has a 
bolt "transform" with executors=1 producing a stream that is comsumed by a bolt 
"ontology" with executors=160.  Everything is configured as shufflegrouping.
With Storm 1.2.3 all of the "ontology" bolts get their fair share of tuples.  
When I run Storm 2.2.0 only the "ontology" bolts that are on the same node as 
the single "transform" bolt get tuples.  
Same cluster - same baseline code - only difference is binding in the new maven 
artifact.
No errors in the logs.  
Any thoughts would be welcome.  Thanks!


  

Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

[Thomas L. Redman]

I have seen this same thing. I sent a query on this list, and. after some time, 
got a response. The issue is reportedly the result of a new feature. I would 
assume this feature is CLEARLY broken, as I had built a test topology that was 
clearly compute-bound, not IO bound, and there were adjustments to change ratio 
of compute/IO. I have not tested this fix, I am too close to release, I just 
rolled back to version 1.2.3.

 From version 2.1.0 forward, not matter how I changed this compute/net IO 
ratio, tuples were not distributed across nodes. Now, I could only reproduce 
this with anchored (acked) tuples. But if you anchored tuples, you could never 
span more than one node, which defeats the purpose of using Storm. Following is 
that email from Kishor Patil identifying the issue (and a way to disable that 
feature!!!):

> From: Kishor Patil 
> Subject: Re: Significant Bug
> Date: October 29, 2020 at 8:07:18 AM CDT
> To: 
> Reply-To: d...@storm.apache.org
> 
> Hello Thomas,
> 
> Apologies for delay in responding here. I tested the topology code provided 
> in storm-issue repo. 
> *only one machine gets peggeg*: Although it appears, his is not a bug. This 
> is related to Locality Awareness. Please refer to 
> https://github.com/apache/storm/blob/master/docs/LocalityAwareness.md
> It appears spout to bolt ratio is 200, so if there are enough bolt's on 
> single node to handle events generated by the spout, it won't send events out 
> to another node unless it runs out of capacity on single node. If you do not 
> like this and want to distribute events evenly, you can try disabling this 
> feature. You can turn off LoadAwareShuffleGrouping by setting 
> topology.disable.loadaware.messaging to true.
> -Kishor
> 
> On 2020/10/28 15:21:54, "Thomas L. Redman"  wrote: 
>> What’s the word on this? I sent this out some time ago, including a GitHub 
>> project that clearly demonstrates the brokenness, yet I have not heard a 
>> word. Is there anybody supporting Storm?
>> 
>>> On Sep 30, 2020, at 9:03 AM, Thomas L. Redman  wrote:
>>> 
>>> I believe I have encountered a significant bug. It seems topologies 
>>> employing anchored tuples do not distribute across multiple nodes, 
>>> regardless of the computation demands of the bolts. It works fine on a 
>>> single node, but when throwing multiple nodes into the mix, only one 
>>> machine gets pegged. When we disable anchoring, it will distribute across 
>>> all nodes just fine, pegging each machine appropriately.
>>> 
>>> This bug manifests from version 2.1 forward. I first encountered this issue 
>>> with my own production cluster on an app that does significant NLP 
>>> computation across hundreds of millions of documents. This topology is 
>>> fairly complex, so I developed a very simple exemplar that demonstrates the 
>>> issue with only one spout and bolt. I pushed this demonstration up to 
>>> github to provide the developers with a mechanism to easily isolate the 
>>> bug, and maybe provide some workaround. I used gradle to build this simple 
>>> topology and software and package the results. This code is well 
>>> documented, so it should be fairly simple to reproduce the issue. I first 
>>> encountered this issue on 3 32 core nodes, but when I started 
>>> experimenting, I set up a test cluster with 8 cores, and then I increased 
>>> each node to 16 cores, and plenty of memory in every case.
>>> 
>>> The topology can be accessed from github at 
>>> https://github.com/cowchipkid/storm-issue.git 
>>> . Please feel free to 
>>> respond to me directory if you have any questions that are beyond the scope 
>>> of this mail list.
>> 
>> 


Hope this helps. Please let me know how this goes, I will upgrade to 2.2.0 
again for my next release.

> On Nov 13, 2020, at 12:53 PM, Michael Giroux  
> wrote:
> 
> Hello, all,
> 
> I have a topology with 16 workers running across 4 nodes.  This topology has 
> a bolt "transform" with executors=1 producing a stream that is comsumed by a 
> bolt "ontology" with executors=160.  Everything is configured as 
> shufflegrouping.
> 
> With Storm 1.2.3 all of the "ontology" bolts get their fair share of tuples.  
> When I run Storm 2.2.0 only the "ontology" bolts that are on the same node as 
> the single "transform" bolt get tuples.  
> 
> Same cluster - same baseline code - only difference is binding in the new 
> maven artifact.
> 
> No errors in the logs.  
> 
> Any thoughts would be welcome.  Thanks!
> 

Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

[Paul Jose]

Hi Michael,

Could it be that the tuples are getting executed much faster on the ontology 
bolt workers than at what rate your transform bolt is executing? Therefore the 
same node would always be ready faster than other workers on other nodes..
If your transform bolt is executing faster than how fast your ontology bolts 
are processing, then I'm not really sure why you're facing this issue..

Best Regards,
Paul

From: Michael Giroux 
Sent: Saturday, November 14, 2020 12:23:43 AM
To: user@storm.apache.org 
Subject: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution 
across cluster

Hello, all,

I have a topology with 16 workers running across 4 nodes.  This topology has a 
bolt "transform" with executors=1 producing a stream that is comsumed by a bolt 
"ontology" with executors=160.  Everything is configured as shufflegrouping.

With Storm 1.2.3 all of the "ontology" bolts get their fair share of tuples.  
When I run Storm 2.2.0 only the "ontology" bolts that are on the same node as 
the single "transform" bolt get tuples.

Same cluster - same baseline code - only difference is binding in the new maven 
artifact.

No errors in the logs.

Any thoughts would be welcome.  Thanks!
---Disclaimer--
 

Views and opinions expressed in this e-mail belong to  their author and do 
not necessarily represent views and opinions  of Ugam. 
Our employees are obliged not to make any defamatory statement or infringe any 
legal right. 
Therefore, Ugam does not accept any responsibility or liability for such 
statements. The content of this email is confidential and intended for the 
recipient specified in message only. It is strictly forbidden to share any part 
of this message with any third party, without a written consent of the sender.
If you have received this message by mistake, please reply to this message and 
follow with its deletion, so that we can ensure such a mistake does not occur 
in the future. 
Warning: Sufficient measures have been taken to scan any presence of viruses 
however the recipient should check this email and any attachments for the 
presence of viruses as full security of the email cannot be ensured despite our 
best efforts.
Therefore, Ugam accepts no liability for any damage inflicted by viewing the 
content of this email.. 

Please do not print this email unless it is necessary. Every unprinted email 
helps the environment. 

Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

[Ravindra Sadaphule]

Unsubscribe

On Fri, Nov 13, 2020 at 10:54 AM Michael Giroux 
wrote:

> Hello, all,
>
> I have a topology with 16 workers running across 4 nodes.  This topology
> has a bolt "transform" with executors=1 producing a stream that is comsumed
> by a bolt "ontology" with executors=160.  Everything is configured as
> shufflegrouping.
>
> With Storm 1.2.3 all of the "ontology" bolts get their fair share of
> tuples.  When I run Storm 2.2.0 only the "ontology" bolts that are on the
> same node as the single "transform" bolt get tuples.
>
> Same cluster - same baseline code - only difference is binding in the new
> maven artifact.
>
> No errors in the logs.
>
> Any thoughts would be welcome.  Thanks!
>
>

-- 
Thanks
Ravindra Sadaphule