Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/4683#issuecomment-75004601
@vanzin The patch is relatively small, but it is "not slightly modified
code". Except some routine code, the critical design is totally different. If
you look into the code in details, you will be aware of it. In another aspect,
you will also find a lot of code in the patch is similar to ats implementation
in Tez, Yarn TimelineClient itself, and Spark related code.
@sryza There are some projects, e.g., Tez, already supporting ATS. It is
hard to wait for other component to be perfect, and I think it is more like
mutual process to improve both components. For example, spark-on-yarn(alpha,
stable, etc) is in similar situation. But it does not prevent it gaining
popularity, and getting more and more stable now.
@andrewor14 The reason I didn't open or attach to a JIRA is that
Spark-1537 has opened and assigned. But the JIRA has opened for almost one
year, and I didn't see progress for long time (please refer to the JIRA
history). This is also the major reason I marked the PR as WIP. In my testing,
the patch can handle most of cases, except some corner cases to be identified.
If you think it is better to have a jira, I will open one.
Folks, thanks for taking time looking at the code. @vanzin If you have
code ready except some caveats from timeline service in yarn. Please go ahead
to send a PR with your patch. I would like to review your patch and don't mind
close mine, so that we can make some progress.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
