[
https://issues.apache.org/jira/browse/AURORA-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864537#comment-15864537
]
Zameer Manji commented on AURORA-1890:
--------------------------------------
The scheduler does the right thing on first pulse. However on failover, any
coordinated updates are immediately sent to BLOCKED_AWAITING_PULSE. This is
because on scheduler startup pulse state is reset to no pulse received. The
code sets the timestamp to the last pulse received to 0L:
{noformat}
synchronized void initializePulseState(IJobUpdate update, JobUpdateStatus
status) {
pulseStates.put(update.getSummary().getKey(), new PulseState(
status,
update.getInstructions().getSettings().getBlockIfNoPulsesAfterMs(),
0L));
}
{noformat}
Would it be ok to set the timestamp to the first event after the most recent
{{BLOCKED_AWAITING_PULSE}}? We know for sure at that point in time that a pulse
was received because of the state transition from {{BLCOKED_AWAITING_PULSE}} to
some other event.
Also could you describe "significant" write volume? I can imagine if the pulse
interval was in the seconds and there are thousands of updates perhaps it would
be too much. However we could prevent excessively small pulse intervals.
> Job Update Pulse History is not durably stored
> ----------------------------------------------
>
> Key: AURORA-1890
> URL: https://issues.apache.org/jira/browse/AURORA-1890
> Project: Aurora
> Issue Type: Bug
> Reporter: Zameer Manji
>
> I have experienced the following problem with pulse updates. To reproduce:
> 1. Create an update with a pulse timeout of 1h
> 2. Send a pulse to get the update going.
> 3. Failover the scheduler immediately after.
> 4. Observe that the update is awaiting another pulse right after the failover.
> This is because the {{JobUpdateControllerImpl}} stores pulse history and
> state in memory in {{PulseHandler}}. On scheduler startup, the pulse state is
> reset to no pulse received.
> We can solve this by durably storing the timestamp of the last pulse received
> in storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
