Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5491#issuecomment-92437781
Question: did you actually run into this problem or is this theoretical?
Because I'd expect a live application to be updating the event log, in
which case its modification time would be changing, and it would then never be
caught by the log cleaner.
If the log is not being updated there's a pretty good chance the
application is dead, and since we don't have shutdown hooks to stop the
SparkContext, the "inprogress" files would be left around indefinitely with
this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
