Github user squito commented on the pull request:
https://github.com/apache/spark/pull/11118#issuecomment-181499494
Reviewers: note this was done primarily by @steveloughran , for now just
posting this as a potential simplification to consider vs.
https://github.com/apache/spark/pull/6935.
Steve -- the main change here is just in `FsHistoryProvider`. I noticed
that `checkForLogs` was going to reload the incomplete apps *anyway* on the
regular scans. So it seemed like there was no point in adding special logic to
do a separate set of reloading. Maybe that reloading is dumb, and we should
change that to separate out finding new apps vs. reloading existing ones, but I
think we might be better off addressing that separately.
It could be that there is some hole in the logic here that I'm not seeing,
but tests pass, so that means we should probably update the tests to address
the hole.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
