Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/8180#issuecomment-137449540
@andrewor14 listening from the sidelines, I think @squito was only pointing
out that you have also noted how hard it is to change the scheduler with
confidence. I don't think he meant to or did imply you are against this change;
clearly the linked JIRA shows you are just favoring more tests as a remedy to
the problem you describe.
There does seem to be some disconnect between SPARK-8987 and @mateiz's view
that there's not much problem changing the scheduler. FWIW it does strike me as
a big change to code that has been a source of subtle bugs in recent memory.
So, maybe a little more nervousness is called for.
We can't stop changing the scheduler and I'm sure that the motivations for
this change are worthy. What's the best way forward -- @squito pointed out some
aspects that make him most nervous; would a few more tests help de-stress? I
tend to respect his nervousness as he takes the brunt of these problems from
big deployments, and getting this working for large deployments reliably is of
course something everyone needs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
