Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4683#issuecomment-75114173
@zhzhan take a look at the JIRA where Marcelo posted some of his code. I'm
also not clear what you and Zhijie are looking for; it seemed like there has
been code and specific YARN issues on offer for a long time. In fact didn't you
reuse some of what @vanzin posted?
But it looks like many of the YARN issues discussed there are resolved as
of 2.6, except at least this one involving developer APIs. Maybe you can work
on that? I can understand the argument that, hey, let's use non-public APIs and
take the risk, but it might be a better use of time to just have it declared
stable then. I'd rather not repeat the loop in
https://issues.apache.org/jira/browse/SPARK-1537
My hesitation is that there is already a lot of build complexity to support
a bunch of Hadoop versions, and we'd have to have yet another type of build for
YARN 2.6+ at least here, and if it changes again in 2.7, another one after
that, just to support ATS, which is a nice-to-have but not critical thing. So I
can sympathize with wanting to get this right, once.
Is that the only issue? I mean, if that were stable, is this PR what we'd
all kind of like to commit? or are we realistically waiting for YARN 2.7, 2.8
before everyone's happy? in which case, yeah, I don't think a `[WIP]` PR makes
sense. You can sure keep a branch / patch / design doc live in the JIRA. Even
`[WIP]` PRs should be for those with reasonable expectation of going in soonish.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
