Rohit Agarwal created SPARK-9924:
------------------------------------
Summary: checkForLogs and cleanLogs are scheduled at fixed rate
and can get piled up
Key: SPARK-9924
URL: https://issues.apache.org/jira/browse/SPARK-9924
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 1.4.0
Reporter: Rohit Agarwal
{{checkForLogs}} and {{cleanLogs}} are scheduled using
{{ScheduledThreadPoolExecutor.scheduleAtFixedRate}}. When their execution takes
more time than the interval at which they are scheduled, they get piled up.
This is a problem on its own but the existence of SPARK-7189 makes it even
worse. Let's say there is an eventLog which takes 15s to parse and which
happens to be the last modified file (that gets reloaded again and again due to
SPARK-7189.) If this file stays the last modified file for, let's say, an hour,
then a lot of executions of that file would have piled up as the default
{{spark.history.fs.update.interval}} is 10s. If there is a new eventLog file
now, it won't show up in the history server ui for a long time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
