Github user squito commented on the pull request:
https://github.com/apache/spark/pull/5636#issuecomment-99141052
Hi Ilya, unfortunately I think this is not doing quite what we want. The
tests aren't quite testing the right thing -- its confusing b/c in the test
environment the `TaskScheduler` is mocked and we are manually doing some of the
stage submission.
Sorry this turned out to be quite complicated in the end -- I think since
we've missed 1.4, we should probably wait till we figure out the concurrent
stage attempt issue, it really makes it a lot harder to fix. I know its
frustrating but I think for now we should just sit on this. (I hope to have a
PR for dealing w/ the concurrent attempts later this week, but I expect that to
also need a lot of scrutiny before it can be merged.)
I'll drop a few notes inline as well on the details of what needs to be
changed, but even with those changes I think the concurrent attempt thing will
still be a blocker.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
