Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/7600#issuecomment-125732553
@dragos I didnt delete any comment! The comments may have been to earlier
threads which are currently hidden due to outdated diffs.
I know that these TestSuiteBase are the helper methods that I have provided
with Spark Streaming. They were done a long time ago and since then we have
learnt quite a bit about the shortcoming of those methods (`runStreams`), and
when to use them and when its cleaner to not use them. The problem with
`runStreams` is that
1. it hides part of the management of life cycle of the context. It starts
and stop the StreamingContext, which give little control when you want to
optimize stuff like.
- Share SparkContexts across StreamingContexts is not possible through
this as the Improves test times if we dont have to spin up SparkContext for
every unit test.
- Tests have unnecessary delays when designed using `runStreams`. This you
see for yourself. With or without `runStreams` you have to use `eventually` to
test your condition. So why wait for the runStreams to complete (especially
when running a real clock) and then test with `eventually`? And how do you
decide how many batches to run in `runStreams`? Is it even important to care
about the number of batches to run considering the test really wants to verify
some other condition being eventually true? Also why go to the complexity of
attaching a `TestOutputDStream` if the test does care about the output? It will
improve test times as well simplify the test if in these cases you just test
with `eventually` alone, without using abstractions that does not really help.
Look at all the newly updated streaming testsuitesl, none of them use
`runStreams`. For a reason. Hope this clarifies why I am trying to move the
code away from `runStreams`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
