GitHub user tedxia opened a pull request:
https://github.com/apache/storm/pull/322
STORM-564: Worker use dynamic port
PR for STORM-564
The main change are:
1. Add worker.dynamic.port to indicate whether worker bind dynamic port or
not;
2. Add node+port->bind-port in Assignment for tell worker's real bind port;
The whole process are:
1. Worker launch receiver, if worker.dynamic.port set as true, receiver
will bind 0, or receiver will bind port specified by supervisor.slots.ports.
2. After receiver launched, set it's real bind port to (:bind-port
worker-data);
3. Set up worker heartbeat thread, add (:bind-port worker-data) to executor
heartbeat info, this will take (:bind-port) to nimbus;
4. Nimbus generate schedule table periodically:
1) worker.dynamic.port set true: when new schedule table and cur
schedule table have common in executor->node+port, nimbus will select the same
part and add the corresponding node+port into node+port->bind-port. Here we
does't add newly scheduled node+port into node+port->bind-port, because until
worker launched receiver and finished it's first heartbeat, nimbus can know the
worker's bind port;
2) worker.dynamic.port set false: we will add all node+port in
ececutor->node+port (come form new schedule table) into node+port->bind-port,
with bind-port is same as port. This make the new version act exactly the same
as older version when set worker.synamic.port as false.
5. Worker read nimbus assignment, and it will connect the worker that
bind-port not null;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tedxia/incubator-storm worker-use-dynamic-port
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/322.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #322
----
commit 4c0b956a48f2d73c49858568d3d60ec78e644b25
Author: xiajun <[email protected]>
Date: 2014-11-12T04:04:34Z
change for worker user dynamic worker
Conflicts:
conf/defaults.yaml
storm-core/src/clj/backtype/storm/cluster.clj
storm-core/src/clj/backtype/storm/daemon/worker.clj
storm-core/src/clj/backtype/storm/messaging/loader.clj
storm-core/src/jvm/backtype/storm/messaging/IConnection.java
storm-core/src/jvm/backtype/storm/messaging/netty/Server.java
storm-dist/binary/src/main/assembly/binary.xml
commit 2a92092ffc064cd7a26c51d572648e9c2ab7d8b3
Author: xiajun <[email protected]>
Date: 2014-11-19T04:57:26Z
add some test code for worker.dynamic.port
Conflicts:
storm-core/test/clj/backtype/storm/cluster_test.clj
storm-core/test/clj/backtype/storm/worker_test.clj
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
