Josh Rosen created SPARK-5079:
---------------------------------
Summary: Detect failed jobs / batches in Spark Streaming unit tests
Key: SPARK-5079
URL: https://issues.apache.org/jira/browse/SPARK-5079
Project: Spark
Issue Type: Bug
Components: Streaming
Reporter: Josh Rosen
Currently, it is possible to write Spark Streaming unit tests where Spark jobs
fail but the streaming tests succeed because we rely on wall-clock time plus
output comparision in order to check whether a test has passed, and hence may
miss cases where errors occurred if they didn't affect these results. We
should strengthen the tests to check that no job failures occurred while
processing batches.
See https://github.com/apache/spark/pull/3832#issuecomment-68580794 for
additional context.
The StreamingTestWaiter in https://github.com/apache/spark/pull/3801 might also
fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
