Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/8153#issuecomment-130777475
In a way I think the underlying issue is more a problem with the aggressive
default polling interval (10 seconds?). But this is a way to make it better.
I think in the (not so distant) future we should investigate using the
recently added inotify-like API in HDFS, to see whether it helps us avoid
polling altogether.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
