[
https://issues.apache.org/jira/browse/STORM-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219121#comment-14219121
]
ASF GitHub Bot commented on STORM-564:
--------------------------------------
GitHub user tedxia opened a pull request:
https://github.com/apache/storm/pull/322
STORM-564: Worker use dynamic port
PR for STORM-564
The main change are:
1. Add worker.dynamic.port to indicate whether worker bind dynamic port or
not;
2. Add node+port->bind-port in Assignment for tell worker's real bind port;
The whole process are:
1. Worker launch receiver, if worker.dynamic.port set as true, receiver
will bind 0, or receiver will bind port specified by supervisor.slots.ports.
2. After receiver launched, set it's real bind port to (:bind-port
worker-data);
3. Set up worker heartbeat thread, add (:bind-port worker-data) to executor
heartbeat info, this will take (:bind-port) to nimbus;
4. Nimbus generate schedule table periodically:
1) worker.dynamic.port set true: when new schedule table and cur
schedule table have common in executor->node+port, nimbus will select the same
part and add the corresponding node+port into node+port->bind-port. Here we
does't add newly scheduled node+port into node+port->bind-port, because until
worker launched receiver and finished it's first heartbeat, nimbus can know the
worker's bind port;
2) worker.dynamic.port set false: we will add all node+port in
ececutor->node+port (come form new schedule table) into node+port->bind-port,
with bind-port is same as port. This make the new version act exactly the same
as older version when set worker.synamic.port as false.
5. Worker read nimbus assignment, and it will connect the worker that
bind-port not null;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tedxia/incubator-storm worker-use-dynamic-port
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/322.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #322
----
commit 4c0b956a48f2d73c49858568d3d60ec78e644b25
Author: xiajun <[email protected]>
Date: 2014-11-12T04:04:34Z
change for worker user dynamic worker
Conflicts:
conf/defaults.yaml
storm-core/src/clj/backtype/storm/cluster.clj
storm-core/src/clj/backtype/storm/daemon/worker.clj
storm-core/src/clj/backtype/storm/messaging/loader.clj
storm-core/src/jvm/backtype/storm/messaging/IConnection.java
storm-core/src/jvm/backtype/storm/messaging/netty/Server.java
storm-dist/binary/src/main/assembly/binary.xml
commit 2a92092ffc064cd7a26c51d572648e9c2ab7d8b3
Author: xiajun <[email protected]>
Date: 2014-11-19T04:57:26Z
add some test code for worker.dynamic.port
Conflicts:
storm-core/test/clj/backtype/storm/cluster_test.clj
storm-core/test/clj/backtype/storm/worker_test.clj
----
> Support worker use dynamic port
> -------------------------------
>
> Key: STORM-564
> URL: https://issues.apache.org/jira/browse/STORM-564
> Project: Apache Storm
> Issue Type: Improvement
> Affects Versions: 0.10.0
> Reporter: xiajun
> Assignee: xiajun
> Priority: Minor
>
> Background: When deploy storm mixed with other services, or deploy storm by
> yarn and mesos or some other scheduling system, worker port conflict is
> really a big problem.
> In order to fix this, we add worker.dynamic.port to indicate whether worker
> bind dynamic port or not.
> When set worker.dynamic.port as true, worker will use port that specified by
> supervisor.slots.ports;
> When set worker.dynamic.port as false, worker will bind 0, and the port that
> specified by supervisor.slots.ports will be nominally port in storm, which
> stand for worker really bind port.
> And when worker launched, worker will report it's really bind port by
> heartbeat, and nimbus send worker's bind port by assignment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
