[
https://issues.apache.org/jira/browse/STORM-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16734943#comment-16734943
]
Stig Rohde Døssing edited comment on STORM-3256 at 1/5/19 4:47 PM:
-------------------------------------------------------------------
You're right. The restriction looks like it has been there at least since Netty
was introduced, and is still there in the current master. Maybe we can change
the message to use an int instead of a short to represent the task id.
If you want to work on this, please let me know and I'll add you to the
contributors list in JIRA. Then you should be able to assign the issue to
yourself.
The person working on this might want to try to account for rolling upgrades,
so the part of the code that reads task messages might need to be able to
handle both int+short and int+int headers.
The part of the code that handles reading the messages is in MessageDecoder. An
off the cuff idea for supporting rolling upgrades here would be adding a
topology level configuration flag that is disabled by default, which makes the
decoder and MessageBatch use 4 bytes for the task id. That way people can
upgrade their clusters while keeping existing topologies running, and when
deploying a topology that needs more than 32k tasks, they can set the flag to
true.
was (Author: srdo):
You're right. The restriction looks like it has been there at least since Netty
was introduced, and is still there in the current master. Maybe we can change
the message to use an int instead of a short to represent the task id.
If you want to work on this, please let me know and I'll add you to the
contributors list in JIRA. Then you should be able to assign the issue to
yourself.
The person working on this might want to try to account for rolling upgrades,
so the part of the code that reads task messages might need to be able to
handle both int+short and int+int headers.
The part of the code that handles reading the messages is in MessageDecoder.
I'm not sure how to support rolling upgrades in this part of the code, but
maybe someone has an idea.
> If all thread counts exceed 32767, the system will generate errors
> ------------------------------------------------------------------
>
> Key: STORM-3256
> URL: https://issues.apache.org/jira/browse/STORM-3256
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-client
> Affects Versions: 1.1.0
> Environment: CentOs 7.1
> Reporter: bb_wolf
> Priority: Blocker
> Labels: patch
>
> If all thread counts exceed 32767, the system will generate errors,
> MessageBatch.java Line141.
>
> Storm systems are used for low latency and a linear increase in the number of
> concurrencies as servers increase.
> If you set up four workers per server, it's easy to achieve 4000 degrees of
> parallelism, that is, 4000 threads per server, then 10 servers will exceed
> 32767.
> Supporting only 32767 threads is a disaster for large-scale computing.
> It is hoped that this function can be improved and repaired as soon as
> possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
