Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/204#issuecomment-39981961
Other then adding a bit of documentation the changes look good to me. I
tested it out and things are working well on spark on yarn.
Note that in the future you will have to be careful about changing the
format of the history files or having the readers handle changes in order keep
backwards compatibility.
Also, the history server will require changes to support kerberos (reading
from keytab file) so that a super user could run the history server and read
from hdfs without worrying about kinit'ing every 24 hours.
I'm also interested to make it work with failed/killed applications. Note
that I tried to force it to work by just touching a APPLICATION_COMPLETE file
bug it fails with "Skipping incomplete application".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
