We are still trying to diagnose our heartbeat issue.
With one of our topologies, workers consistently stop heart beating after a
variable amount of time.
(On the worker, CPU is fine, memory is fine.)
To help diagnose, we dropped some debugs statements into the timer.clj, and
we see the timer loop seize up.
The last line of output we see is:
³Doing loop for (timer_27)²
With the following code:
(while @active
(try
(log-warn "Doing loop for (timer_27) ")
(let [[time-millis _ _ :as elem] (locking lock
(.peek queue))]
(if (and elem (>= (current-time-millis)
time-millis))
;; imperative to not run the function inside
the timer lock
;; otherwise, it's possible to deadlock if
function deals with other locks
;; (like the submit lock)
(let [afn (locking lock (second (.poll
queue)))]
(log-warn "Doing timer if stm (timer_35) "
(pr-str afn))
And then the output to the log halts.
Any ideas? (are we maybe hitting the dead lock mentioned in the comments?)
-brian
---
Brian O'Neill
Chief Architect
Health Market Science
The Science of Better Results
2700 Horizon Drive King of Prussia, PA 19406
M: 215.588.6024 @boneill42 <http://www.twitter.com/boneill42>
healthmarketscience.com
This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or the
person responsible to deliver it to the intended recipient, please contact
the sender at the email above and delete this email and any attachments and
destroy any copies thereof. Any review, retransmission, dissemination,
copying or other use of, or taking any action in reliance upon, this
information by persons or entities other than the intended recipient is
strictly prohibited.
