[
https://issues.apache.org/jira/browse/SPARK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-9924:
-----------------------------------
Assignee: (was: Apache Spark)
> checkForLogs and cleanLogs are scheduled at fixed rate and can get piled up
> ---------------------------------------------------------------------------
>
> Key: SPARK-9924
> URL: https://issues.apache.org/jira/browse/SPARK-9924
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.0
> Reporter: Rohit Agarwal
>
> {{checkForLogs}} and {{cleanLogs}} are scheduled using
> {{ScheduledThreadPoolExecutor.scheduleAtFixedRate}}. When their execution
> takes more time than the interval at which they are scheduled, they get piled
> up.
> This is a problem on its own but the existence of SPARK-7189 makes it even
> worse. Let's say there is an eventLog which takes 15s to parse and which
> happens to be the last modified file (that gets reloaded again and again due
> to SPARK-7189.) If this file stays the last modified file for, let's say, an
> hour, then a lot of executions of that file would have piled up as the
> default {{spark.history.fs.update.interval}} is 10s. If there is a new
> eventLog file now, it won't show up in the history server ui for a long time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
