GitHub user kevinpetersavage reopened a pull request:
https://github.com/apache/spark/pull/4957
Spark 5659 Flaky test: o.a.s.streaming.ReceiverSuite.block
I made two changes to fix this. First I replaced the RateLimiter code with
guavas rate limiter which seemed to behave better. I added an extra couple of
tests for this class.
Secondly I simplified the assertions in the test so that we send a fixed
number of messages and assert on the wait time (rather than the other way
round). I also completely removed the last assertion which check for roughly
equal batch sizes because I think it is not valid. The JVM might just be doing
other things, stopping this being predictable and indeed in my testing I found
that while we expected ~10 messages per block we sometimes got things like 50
messages in one block.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kevinpetersavage/spark SPARK-5659
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4957.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4957
----
commit c39211bd6858b82df5f8e2551a7bbaaeb78d88f7
Author: Kevin Savage <[email protected]>
Date: 2015-03-08T00:58:39Z
SPARK-5659: Fix flaky test o.a.s.streaming.ReceiverSuite.block. Simplifies
RateLimiter and removes assertion that is not valid.
commit b11b4e13d085b6103f27bfac25eba62244023807
Author: Kevin Savage <[email protected]>
Date: 2015-03-09T22:01:10Z
Merge branch 'master' into SPARK-5659
commit 10c502c241988df5db0e42dffd7510fb53f4422f
Author: Kevin Savage <[email protected]>
Date: 2015-03-12T12:19:32Z
SPARK-5659: limit the number of messages in a block
commit ac8a07ac6e6b6f87ce43f8713a36bc14c005aa5d
Author: Kevin Savage <[email protected]>
Date: 2015-03-12T12:27:16Z
SPARK-5659: implement suggested changes
commit 158cd89d450112d16885ca4f11dc98da365fa339
Author: Kevin Savage <[email protected]>
Date: 2015-03-12T15:51:04Z
SPARK-5659: fix line being to long
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
