Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7572#issuecomment-123742771
agree that (1) this is not a regression, and (2) that the lack of bug
reports is more from confusion than not encountering it. I'm nearly certain
that I have run into this (or maybe it was SPARK-8029 or SPARK-5259 ... plus a
little SPARK-5945 mixed in) many times in the past but simply been too confused
/ busy to understand what was going on, and assumed it must have been something
we were doing wrong. We also have bug reports from customers where they are
running into this. But even sophisticated customers that have made other
requests that we work on SPARK-XYZ have never realized this was the issue they
were hitting.
FWIW, I'm going to be thinking very carefully about backporting that whole
set of scheduler changes to versions of CDH that are based on pre-1.5. I'm
definitely nervous about it, but it might be necessary just because of how
important these fixes are. Frankly the unit tests are not nearly enough to
gain confidence in the scheduler, so I'll need to do a lot more testing before
I'm comfortable with that. Not that it needs to effect our decision on
backporting inside spark itself, just thought it may be a useful data point.
But I suppose if I try to look at this objectively, this isn't worth
breaking our policy of what we backport. Perhaps we should consider changing
that release model as we get mor stability, but that is a separate discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
