Github user liyezhang556520 commented on the pull request:
https://github.com/apache/spark/pull/5886#issuecomment-101308561
@vanzin , the current implementation will make SPARK-7189 worse, what about
introduce a hashMap to maintain the filename, modifiedTime of each file, file
size, say mutable.HashMap[String, Long, Long], it can not only handle the
modification case and also rename/delete cases. Since each file's modification
time is maintained, this can both solve the problem of the race condition in
this issue and also solve SPARK-7189? And it will only introduce extra memory
with size of the hashMap size.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
