Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2471#issuecomment-57666110
> a system approach means using something like logrotate or a cleaner
process that's run from cron.
The only thing you can really use system utilities for is cron, which is
the least important part of this change. Really, this is not an expensive
process that will bring down the HDFS server, and it's scheduled to run at very
long intervals. The constant polling for new logs is orders of magnitude more
disruptive than this cleanup thread.
AFAIK, logrotate doesn't work on HDFS. Now you'd be asking for people to
set us the NFS bridge or even fuse-hdfs just to clean up Spark event log files.
Finally, Spark theoretically supports Windows. This is a simple way to
achieve compatibility with that. And it doesn't require people to set things up
outside of their Spark ecosystem, meaning it's easier to maintain.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
