Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1222#issuecomment-50552144
Yeah, encoding attributes in the file name is not optimal. But I was trying
to avoid having to write a separate resource containing the metadata, to avoid
too many round trips to HDFS. I was thinking that extended attributes (HDFS
2.5) could be a solution, but they still require a separate call to HDFS to
retrieve them, and we still need to support older versions. I need to think
more about this to see what a good solution would be here.
Regarding FileLogger, I removed it because it only had one user, and the
only functionality it really provided on top of the raw streams (the "new file
with incremented index" method) is not needed anymore. So I didn't see the need
to keep it around - it's not really providing anything useful anymore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
