GitHub user dragos opened a pull request:
https://github.com/apache/spark/pull/7600
[WIP][SPARK-8977][Streaming] Defines the RateEstimator interface, and
impements the RateController
Based on #7471.
- [ ] add a test that exercises the publish path from driver to receiver
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/typesafehub/spark
topic/streaming-bp/rate-controller
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7600.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7600
----
commit 4721c7d9394b7917b2be8fb7df5e4eb1c31d68df
Author: François Garillot <[email protected]>
Date: 2015-07-13T11:26:57Z
[SPARK-8975][Streaming] Add a mechanism to send a new rate from the driver
to the block generator
commit d15de422b973a020d5aa9035016c1274262631fb
Author: François Garillot <[email protected]>
Date: 2015-07-15T11:14:31Z
[SPARK-8975][Streaming] Adds Ratelimiter unit tests w.r.t.
spark.streaming.receiver.maxRate
commit 6369b305b87f812e0bbb6a37e2a1d338c1e1cb26
Author: Iulian Dragos <[email protected]>
Date: 2015-07-17T09:20:38Z
Merge pull request #15 from huitseeker/SPARK-8975
[SPARK-8975][Streaming] Adds a mechanism to send a new rate from the driver
to the block generator
commit cd1397d141eda98ded62491c0f2d90a2b47e56c5
Author: Iulian Dragos <[email protected]>
Date: 2015-07-17T14:38:10Z
Add a test for the propagation of a new rate limit from driver to receivers.
commit 261a05128ec1e1e055c62a6afd44fef39fb711c1
Author: Iulian Dragos <[email protected]>
Date: 2015-07-20T12:28:46Z
- removed field to hold the current rate limit in rate limiter
- made rate limit a Long and default to Long.MaxValue (consequence of the
above)
- removed custom `waitUntil` and replaced it by `eventually`
commit 0c51959c9315f63bc80a7ff5b716f48f907b1152
Author: Iulian Dragos <[email protected]>
Date: 2015-07-20T14:40:01Z
Added a few tests that measure the receiverâs rate.
As I mentioned before, I donât think this is a great idea:
- such tests are flaky (original test in ReceiverSuite was ignored for
that reason)
- Guavaâs code has its own test suite, so we can assume it implements
`setRate` correctly
I noticed one flaky failure in about 10 runs on my machine (receiver got
1 message less than the lower bound, which is within 5% of the nominal
rate).
commit 210f495fff34f25caaba41a8db720c1e3a63fa95
Author: Iulian Dragos <[email protected]>
Date: 2015-07-21T12:29:04Z
Revert "Added a few tests that measure the receiverâs rate."
This reverts commit 0c51959c9315f63bc80a7ff5b716f48f907b1152.
commit 162d9e598040b5b2bad36fa0f7139a99df95e79d
Author: Iulian Dragos <[email protected]>
Date: 2015-07-21T13:06:13Z
Use Reflection for accessing truly private `executor` method and
use the listener bus to know when receivers have registered (`onStart`
is called before receivers have registered, leading to flaky behavior).
commit 8941cf91b03aa7835a78bc756bee1f32cb7bb1d8
Author: Iulian Dragos <[email protected]>
Date: 2015-07-22T14:45:57Z
Renames and other nitpicks.
commit d32ca3697ab18ba8db9905c81d57559bf8472195
Author: François Garillot <[email protected]>
Date: 2015-07-14T11:11:01Z
[SPARK-8977][Streaming] Defines the RateEstimator interface, and implements
the ReceiverRateController
commit 8b084921fc90ad133a5231e00fccbdbbd69ef84b
Author: Iulian Dragos <[email protected]>
Date: 2015-07-22T21:13:24Z
Various style changes and a first test for the rate controller.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
