GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/9913
[SPARK-7889] [Core] WiP History Caching: xu tingjun's patch against master
This is @XuTingjun's patch (#6545) reapplied to trunk with a bit of cleanup
(the probe in the history provider is now `isComplete(String appId,
Option[String] attemptId`; the conditions around its use and is internals
slightly tweaked.
Looking at this code now, I do agree the history provider should have a say
in the state of the attempts, I'm not sure if this is the right approach.
Having 3x different maps to track app state (including two that never get
cleaned up) isn't ideal, and if it were to be so, then the two new ones would
have to be thread safe, so as to handle the scenario of "multiple requests
coming in at the same time"
I'd rather retain the history server's information alongside the cached
app, (as my patch does), though that still leaves the question of how to probe
for a updated version and then reload it, especially in a way that is thread
safe.
Oh, and testing, obviously.
(once this PR has been tested I'm going to close it; it's here as reference)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/steveloughran/spark
history/SPARK-7889-XuTingjun
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9913.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9913
----
commit 1add1ec9dd315e5b2cf9d2a77e95f7e47b3bf5c3
Author: Steve Loughran <[email protected]>
Date: 2015-11-23T18:40:35Z
SPARK-7889 apply XuTingjun's patch to trunk
commit 865808e4f555396b91422780dd6267a0457d9d43
Author: Steve Loughran <[email protected]>
Date: 2015-11-23T18:55:38Z
SCALA-7889
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
