[
https://issues.apache.org/jira/browse/AURORA-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304424#comment-14304424
]
Bill Farner edited comment on AURORA-1096 at 4/17/15 10:21 PM:
---------------------------------------------------------------
Clarification:
A cap on {{JobUpdateEvent}} s can only be applied dynamically. When the limit
is met, we should transition the update to the {{ERROR}} state. We can't
enforce this at update start time because it is effectively unbounded (a user
can pause/resume forever).
The cap on {{JobInstanceUpdateEvent}} should be imposed at job update
submission time, by calculating the maximum number of instance events permitted
by the configuration.
was (Author: wfarner):
Clarification:
A cap on {{JobUpdateEvent}}s can only be applied dynamically. When the limit
is met, we should transition the update to the {{ERROR}} state. We can't
enforce this at update start time because it is effectively unbounded (a user
can pause/resume forever).
The cap on {{JobInstanceUpdateEvent}} should be imposed at job update
submission time, by calculating the maximum number of instance events permitted
by the configuration.
> Scheduler updater should limit the number of job/instance events
> ----------------------------------------------------------------
>
> Key: AURORA-1096
> URL: https://issues.apache.org/jira/browse/AURORA-1096
> Project: Aurora
> Issue Type: Story
> Components: Scheduler
> Reporter: Maxim Khutornenko
>
> Large/flapping scheduler job updates may generate too many events in the
> update store. The update settings are fully controlled by the user and there
> is a potential for a misconfigured job update to completely overwhelm our
> in-memory DB storage with job update instance events.
> For example, a large flapping update with {{max_per_shard_failures}} and
> {{max_total_failures}} set to max INT when left unattended can quickly
> consume all available RAM and kill the scheduler. A manual cleanup of the
> scheduler log would be needed to bring the scheduler up.
> This can be especially relevant with the introduction of update heartbeats
> (AURORA-690) that can further exacerbate the problem (e.g. when
> {{blockIfNoPulseAfterMs}} set too low wrt the external service pulse rate).
> We need to cap the max per-job lifetime count of {{JobUpdateEvent}} and
> {{JobInstanceUpdateEvent}} instances. A nice bonus would be providing a hint
> in the UI when the event sequence is cut off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
