Github user mattf commented on the pull request:
https://github.com/apache/spark/pull/2471#issuecomment-59191797
> Well, good luck with adding something like that to HDFS... that is not
the responsibility of filesystems.
just so we're on the same page, i'm not advocating adding this
functionality to HDFS. i believe it should be separate functionality that
doesn't live in a spark process, as it's an operational activity.
> a directory that is completely managed within Spark (the event log
directory) shouldn't also be cleaned up by Spark
we can agree to disagree. you can view a trace log as allocating some
amount of disk space that you then have to manage, similar to memory
allocations in a program. however, doing that management is more involved than
periodically rotating.
if you're in nyc for strata we should grab a beer and debate the finer
points of resource management.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
