[
https://issues.apache.org/jira/browse/SPARK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin updated SPARK-9924:
----------------------------------
Fix Version/s: 1.5.1
> checkForLogs and cleanLogs are scheduled at fixed rate and can get piled up
> ---------------------------------------------------------------------------
>
> Key: SPARK-9924
> URL: https://issues.apache.org/jira/browse/SPARK-9924
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.0
> Reporter: Rohit Agarwal
> Assignee: Rohit Agarwal
> Fix For: 1.6.0, 1.5.1
>
>
> {{checkForLogs}} and {{cleanLogs}} are scheduled using
> {{ScheduledThreadPoolExecutor.scheduleAtFixedRate}}. When their execution
> takes more time than the interval at which they are scheduled, they get piled
> up.
> This is a problem on its own but the existence of SPARK-7189 makes it even
> worse. Let's say there is an eventLog which takes 15s to parse and which
> happens to be the last modified file (that gets reloaded again and again due
> to SPARK-7189.) If this file stays the last modified file for, let's say, an
> hour, then a lot of executions of that file would have piled up as the
> default {{spark.history.fs.update.interval}} is 10s. If there is a new
> eventLog file now, it won't show up in the history server ui for a long time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
