GitHub user strongh opened a pull request:
https://github.com/apache/incubator-storm/pull/197
Memoize local hostname lookup in executor
This change adds a memoized version of `util/local-hostname` using
`clojure.core/memoize`, and uses the memoized version in the executor.
This issue is motived by behavior we've experienced @yieldbot where
intermittent DNS problems. During these intermittent network problems (thanks
EC2), the most common exceptions are `UnknownHostException` like the on with
the stack trace below:
```
java.lang.RuntimeException: java.net.UnknownHostException: node-3.storm:
node-3.storm at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at
clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.UnknownHostException: node-3.storm: node-3.storm at
java.net.InetAddress.getLocalHost(InetAddress.java:1374) at
backtype.storm.daemon.executor$metrics_tick.invoke(executor.clj:280) at
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:620)
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:402)
at backtype.storm.disruptor$clojure_handler$reify
__745.onEvent(disruptor.clj:58) at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
... 6 more
```
The hope is that this change can reduce the impact of intermittent network
issues. I should note that we use a positive `sun.net.inetaddr.ttl`; it's
possible that most users would not experience this problem since the JVM's
default may be to cache indefinitely (e.g. for sun JVM w/o security manager).
We configure `sun.net.inetaddr.ttl` primarily to allow replacement of nodes
without needing to restart across the cluster, which I believe is an
operational strategy storm should support.
This change was suggested in the [mailing
list](http://mail-archives.apache.org/mod_mbox/storm-dev/201404.mbox/%3CCA+-fM0vuEy1XgCYJ=0aN26=pf9ijg74f_gffi7branjam9x...@mail.gmail.com%3E)
back in April. It sounds like @jnioche has seen similar problems.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/strongh/incubator-storm
memoize-local-hostname-metrics-tick
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-storm/pull/197.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #197
----
commit a5c97d79e0e809a43b962a8a6753f376e8bc020f
Author: Homer Strong <[email protected]>
Date: 2014-07-20T19:39:20Z
add memoized version of local-hostname in util ns
commit 28de7e511340d6c134f2eac039dc054a0107cba4
Author: Homer Strong <[email protected]>
Date: 2014-07-20T19:39:40Z
use util/memoized-local-hostname in executor
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
