Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6545#issuecomment-158969374
...I'm looking at this and my PR; I think they are differently
interdependent.
1. This patch adds a new method to `ApplicationHistoryProvider` for the
history provider to give its status on an application.
1. plus an implementation in the `FsHistoryProvider` to choose the flag
based on its knowledge of the loaded path and whether the file came has the
{{.".inprogress"}} extension
1. The web UI is registered in `SparkUI` such that hitting paths in the UI
can trigger a reload.
There's some extra map datastructures added in HistoryServer and
FsHistoryProvider to aid this.
In comparison, my pull request #6935, does all its work in a pulled out
ApplicationCache, which adds a completed flag and a timestamp, the goal being
that a refresh is triggered if !completed and now > timestamp + interval.
But that PR doesn't: join up the web UI, or let the providers declare that
an app has completed *except by a complete reload of their UI*
I think the things can be merged; I'd have mine being able to query
(indirectly) the history provider to see if the app has completed. or even
better: if it has changed since the last reload. That would eliminate wasteful
reloads. How to achieve this? either the cache calls through to the history
provider, or the history provider itself provides some callback for the cache
entries to invoke.
There's one more option to consider here. Why do we actually
replay-then-pause in-progress applications? Why not have a thread continually
loading them and forwarding events until they complete? In this mode, there's
no need to reload the cache, replay events....the UI just stays up to date.
I'll cover that on my PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
