Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5886#issuecomment-100379581
> This isn't an issue that affects app correctness right?
It depends on what you're calling "app correctness" here. Isn't missing an
updated / renamed log a correctness issue?
> logs collection fails entirely because touching a file fails
True, but that should only cause one poll to fail. It shouldn't cause the
HS to go down. If the HS can't persistently write to HDFS, that means some
configuration is wrong, since the history directory is expected to be
world-writable.
> Is it worth even trying to address this?
I think it's worth it if it can be done without harming other parts of the
HS (like requiring way more memory than before). I think the latest approach is
close; if some code is added to address SPARK-7189, then it would be ready to
go.
The issue raise by this PR should be a really, really rare situation though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
