[
https://issues.apache.org/jira/browse/SPARK-28869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17458931#comment-17458931
]
muhong commented on SPARK-28869:
--------------------------------
i foud a problem, the thrift server create event log file(.inprogress file
create at init), and history server clean the application event log file
according size and modtime. so there is a potential problem under this situation
*if the thrift server accept no quest long time(longer than time config by
spark.history.fs.cleaner.maxAge), the history server will clean the
applicaiton log [directory] with the inprogress file; after clean the thrift
server accept a lot of request ,and will generate new event log directory
without inprogress status file, and the director will never be clean by history
server because it not contain status file. this will leads spack leak*
i think whenever create new log file , need to check wether the status file is
exist, if not create it
last i think extra function need add, like log4j the compact file stii need to
be clean after a period(config by user)，so ，long run spark service like thrift
server‘s event log file space can be limit in a config size
> Roll over event log files
> -------------------------
>
> Key: SPARK-28869
> URL: https://issues.apache.org/jira/browse/SPARK-28869
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Fix For: 3.0.0
>
>
> This issue tracks the effort on rolling over event log files in driver and
> let SHS replay the multiple event log files correctly.
> This issue doesn't deal with overall size of event log, as well as no
> guarantee when deleting old event log files.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
