[
https://issues.apache.org/jira/browse/STORM-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006177#comment-15006177
]
ASF GitHub Bot commented on STORM-956:
--------------------------------------
Github user bastiliu commented on the pull request:
https://github.com/apache/storm/pull/647#issuecomment-156900352
I think it is better to raise a warning intead of stopping to update worker
heartbeat. There are some topologys that might require to block at
execute()/nextTuple() to wait some essential initialization (e.g. establish
connection to database..).
For the trigger mechanism of hanging on problem, it is a potential risk
that worker might be restart unexpectedly due to heartbeat timeout. Because we
consume a batch of tuples each time. Even through the time of processing one
tuple is short, the time of processing the batch could cause the timeout.
> When the execute() or nextTuple() hang on external resources, stop the
> Worker's heartbeat
> -----------------------------------------------------------------------------------------
>
> Key: STORM-956
> URL: https://issues.apache.org/jira/browse/STORM-956
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Chuanlei Ni
> Assignee: Chuanlei Ni
> Priority: Minor
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> Sometimes the work threads produced by mk-threads in executor.clj hang on
> external resources or other unknown reasons. This makes the workers stop
> processing the tuples. I think it is better to kill this worker to resolve
> the "hang". I plan to :
> 1. like `setup-ticks`, send a system-tick to receive-queue
> 2. the tuple-action-fn deal with this system-tick and remember the time that
> processes this tuple in the executor-data
> 3. when worker do local heartbeat, check the time the executor writes to
> executor-data. If the time is long from current (for example, 3 minutes), the
> worker does not do the heartbeat. So the supervisor could deal with this
> problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
