Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/7600#issuecomment-125747786
1. Cannot deprecate TestSuiteBase as it is still perfectly suited for
testsuites like `BasicOperationsSuite` where the exact per-batch output needs
to be verified. Its suitable for some use cases, just not all usecase,
especially not for those where the output is not important.
2. I visited the new tests in ReceiverTrackerSuite again
(a)"rate controller publishes updates": No need for runStreams on exact
number of batches, all you need is `eventually { publishCalls > 0 }`
(b) "publish rates reach receivers": No need for runStreams on exact number
of batches, all you need is `eventually {dstream.getCurrentRateLimit ===
Some(200) }`
(c) You are trying to run 4 batches and expect there will 3 observed rates
immediately as the 4 batches complete. This CANNOT be guaranteed. Thread.sleep
cannot guarantee that you observe the rate limits at the right times so that
you observe all the different rates. See comments there, I have suggest how to
make this non-flaky. Again, does not require runStreams with exact number of
batches.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
