Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/718#issuecomment-45123904
Hi @pwendell,
First, thanks for taking the time to look into this.
While it's true that this change mixes a few arguably unrelated things, I'm
a little worried about breaking it up further. Mainly because github's pull
request feature doesn't seem to handle incremental changes well (see my
previous comment,
https://github.com/apache/spark/pull/718#issuecomment-45023292). If there's a
way I'm not aware of that allows me to send multiple, related PRs through
github, I'll gladly make the changes. Otherwise, I'd rather keep this PR as is
since I already have more work that depends on it, and breaking it down would
just slow down the whole process even more.
About (3), note that one of the goals of the enhancements, as I mentioned
in the Jira issue, is to prepare the field for SPARK-1537. Also, the
refactoring allows for easier unit testing (which I added as part of my related
change mentioned in my above comment). I'm not worried about having to change
the interface later - these are internal APIs, after all. But I really would
like to have the UI handling separate from the history backend.
About (4), the main thing is to allow arbitrary options to be set more
easily. I'm not wedded to the idea of a new command line argument (see other
comments), but I think we should at least add support for reading config files
(not just in the History Server, but also other daemons). I think that more
closely matches how people generally deploy daemons.
On a side note, just in case github doesn't support the "related PRs" thing
I tried to explain, I'd suggest you guys take a look at something like
ReviewBoard. RB makes that pretty painless, since it's not necessarily tied to
github branches. It also handles things like renaming files more cleanly (when
tied to a git repo). It would probably make merging changes a little more
difficult, but I'm sure that could be made better with some scripts. In any
case, I'm really looking for suggestions about how to handle these related PRs
- sending them one at a time and having to wait until they are committed before
sending the next one really slows things considerably, and prevents people from
seeing the "big picture" when work spans several changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
