Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16924
In `FsHistoryProvider`, since there is a check for file size, I think it is
designed to find updated logs from running applications?
https://github.com/apache/spark/blob/e26dac5feb02033f980b1e69c9b0ff50869b6f9e/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L331
Btw, in `EventLoggingListener`, if it writes to local files, seems to me
the file length will be updated and so `FsHistoryProvider` can find the updated
logs.
Seems this looks reasonable to make the non local fs functionally
consistent with local fs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
