Github user squito commented on the issue:
https://github.com/apache/spark/pull/16620
Hi @jinxing64
sorry to go back and forth on this numerous times -- I think I have another
alternative, see https://github.com/squito/spark/tree/SPARK-19263_alternate
Its most of your changes but with one main difference: when we encounter
the condition where there are no pending partitions, but there is an active
taskset -- we just mark that taskset as inactive and continue as before
https://github.com/squito/spark/commit/bec061c8486a681dc16e8b92e553f79e486924e9.
I think this makes it easier to follow, as there are fewer states to keep
track of. It also can potentially improve performance, since you may submit
downstream stages more quickly, rather than waiting for all tasks in the active
taskset to complete. I also think it fixes a bug in your version with
mapStageJobs (I'll point it out in the code).
This passes all tests in `o.a.s.scheduler.*`, including your new test case.
(I did come across a race in `ScheduleIntegrationSuite` which I fixed
https://github.com/squito/spark/commit/9125e6738269df4e0d7e6292726bad2a294c86c0
not directly related to these changes).
Do you see any problems w/ that approach?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
