Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/6545#issuecomment-112366005
1. I can do the test as part of SPARK-1537; indeed, I already have it
replicated, I've just turned that test off. That code already has everything
needed (listener for spark events, event publishing, fetching of events,
spinning up the history server and GET of the relevant pages).
2. I'd add the test to
[IncompleteSparkUISuite](https://github.com/steveloughran/spark/blob/stevel/feature/SPARK-1537-ATS/yarn/history/src/test/scala/org/apache/spark/deploy/history/yarn/integration/IncompleteSparkUISuite.scala),
which already walks an app through its lifecycle and verifies that the GET
/?incomplete=true listing changes state. All it needs is an extra {{GET
/history/appId}} to verify that its changed state too.
3. My proposed patch in SPARK-8275 was a lot simpler. All I was doing there
was, on a {{GET /history/appId}} was looking in the cache, and if incomplete,
check that the entry hadn't been in the cache longer than an expiry time. This
would ensure that repeated GETs within a short period of time wouldn't have any
performance hit, especially with multiple viewers, but the caching wouldn't be
eternal. After all, the history server already discards old entries once the
cache gets full.
Accordingly, why don't I implement my simple discard algorithm and the
test? If that isn't enough we can look at this code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
