GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/15250
[SPARK-17676][CORE] FsHistoryProvider should ignore hidden files
## What changes were proposed in this pull request?
FsHistoryProvider was writing a hidden file (to check the fs's clock).
Even though it deleted the file immediately, sometimes another thread
would try to scan the files on the fs in-between, and then there would
be an error msg logged which was very misleading for the end-user.
(The logged error was harmless, though.)
## How was this patch tested?
I added one unit test, but to be clear, that test was passing before. The
actual change in behavior in that test is just logging (after the change, there
is no more logged error), which I just manually verified.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-17676
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15250.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15250
----
commit b125f2f2761ae81abd2c2c049666fc1b0ca51e0c
Author: Imran Rashid <[email protected]>
Date: 2016-09-27T02:56:00Z
[SPARK-17676][CORE] FsHistoryProvider should ignore hidden files
FsHistoryProvider was writing a hidden file (to check the fs's clock).
Even though it deleted the file immediately, sometimes another thread
would try to scan the files on the fs in-between, and then there would
be an error msg logged which was very misleading for the end-user.
(The logged error was harmless, though.)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
