Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/8744#issuecomment-144080509
Sorry I haven't had time to follow all of these changes, which version of
ATS are you testing this on and at what scale? I'm assuming ATS v1 since v2
still in devel.
We have been using ATS with tez and it can't handle nearly the load we need
it to. With Spark I see cases where that just gets worse if you have longer
running iterative things. I have seen history files in the 8GB size on spark
before.
I thought the ATS v2 was going to try to keep api compatibility but don't
think its guaranteed. Obviously it doesn't hurt to get framework in place and
I assume it will work for smaller clusters/# jobs. Have you looked at ats v2
at all to know if the changes to go to it would be fairly minimal?
Mostly just curious where this is at as I'm seeing a bunch of small issues
with the Spark history server.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
