[
https://issues.apache.org/jira/browse/SPARK-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or updated SPARK-10372:
------------------------------
Summary: Add end-to-end tests for the scheduling code (was: Add end-to-end
tests for the scheduler)
> Add end-to-end tests for the scheduling code
> --------------------------------------------
>
> Key: SPARK-10372
> URL: https://issues.apache.org/jira/browse/SPARK-10372
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 1.5.0
> Reporter: Imran Rashid
> Assignee: Imran Rashid
> Fix For: 1.6.0
>
>
> The current testing framework for the scheduler only tests individual classes
> in isolation: {{DAGSchedulerSuite}}, {{TaskSchedulerImplSuite}}, etc. Of
> course that is useful, but we are missing tests which cover the interaction
> between these components. We also have larger tests which run entire spark
> jobs, but that doesn't allow fine grained control of failures for verifying
> spark's fault-tolerance.
> Adding a framework for testing the scheduler as a whole will:
> 1. Allow testing bugs which involve the interaction between multiple parts of
> the scheduler, eg. SPARK-10370
> 2. Greater confidence in refactoring the scheduler as a whole. Given the
> tight coordination between the components its hard to consider any
> refactoring, since it would be unlikely to be covered by any tests.
> 3. Make it easier to increase test coverage. Writing tests for the
> {{DAGScheduler}} now requires intimate knowledge of exactly how the
> components fit together -- a lot of work goes into mimicking the appropriate
> behavior of the other components. Furthermore, it makes the tests harder to
> understand for the un-initiated -- which parts are simulating some condition
> of an external system (eg., losing an executor), and which parts are just
> interaction with other parts of the scheduler (eg., task resubmission)?
> These tests will allow to just work at the level of the interaction w/ the
> executors -- tasks complete, tasks fail, executors are lost, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
