[
https://issues.apache.org/jira/browse/SPARK-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010413#comment-14010413
]
Tathagata Das edited comment on SPARK-1940 at 6/4/14 1:50 AM:
--------------------------------------------------------------
This is solved in https://github.com/apache/spark/pull/895
This PR solves this by implementing a simple `RollingFileAppender` within Spark
(disabled by default). When enabled (using configuration parameter
`spark.executor.rollingLogs.enabled`), the logs can get rolled over either by
time interval (set with `spark.executor.rollingLogs.interval`, set to daily by
default), or by size of logs (set with `spark.executor.rollingLogs.size`).
Finally, old logs can be automatically deleted by specifying how many of the
latest log files to keep (set with `spark.executor.rollingLogs.keepLastN`). The
web UI can show the logs across the rolled over files.
was (Author: tdas):
This is solved in https://github.com/apache/spark/pull/895
This PR solves this by implementing a simple RollingFileAppender within Spark
(disabled by default). When enabled (using configuration parameter
spark.executor.rollingLogs.enabled), the logs can get rolled over by time
interval (set with spark.executor.rollingLogs.cleanupTtl, set to daily by
default). Old logs (older than 7 days, by default) will get deleted
automatically. The web UI can show the logs across the rolled over files.
> Enable rolling of executor logs (stdout / stderr)
> -------------------------------------------------
>
> Key: SPARK-1940
> URL: https://issues.apache.org/jira/browse/SPARK-1940
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Tathagata Das
> Assignee: Tathagata Das
>
> Currently, in the default log4j configuration, all the executor logs get sent
> to the file <code>[executor-working-dir]/stderr</code>. This does not all log
> files to be rolled, so old logs cannot be removed.
> Using log4j RollingFileAppender allows log4j logs to be rolled, but all the
> logs get sent to a different set of files, other than the files
> <code>stdout</code> and <code>stderr</code> . So the logs are not visible in
> the Spark web UI any more as Spark web UI only reads the files
> <code>stdout</code> and <code>stderr</code>. Furthermore, it still does not
> allow the stdout and stderr to be cleared periodically in case a large amount
> of stuff gets written to them (e.g. by explicit println inside map function).
> Solving this requires rolling of the logs in such a way that Spark web UI is
> aware of it and can retrieve the logs across the rolled-over files.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
